Repository: stfc/fparser

Issue:
fparser.api.parse fails for long comment blocks
When parsing a kernel that has a comment header longer than 161 lines PSyclone reports the incorrect Fortran error, e.g.
```
"Parse Error: Failed to parse kernel code 'test_fparser_kernel_mod.f90'. Is the Fortran correct?"
```
The error is raised in the [`psyclone.parse.kernel.get_kernel_parse_tree`](https://github.com/stfc/PSyclone/blob/master/src/psyclone/parse/kernel.py#L146) function. However, this is actually an fparser failure but that is not visible from PSyclone as the fparser logging is [disabled](https://github.com/stfc/PSyclone/blob/master/src/psyclone/parse/kernel.py#L160).

`psyclone.parse.kernel.get_kernel_parse_tree` calls [`fparser.api.parse`](https://github.com/stfc/fparser/blob/master/src/fparser/api.py#L150) function. This function, in turn, calls the [`fparser.one.parsefortran.FortranParser.parse`](https://github.com/stfc/fparser/blob/master/src/fparser/one/parsefortran.py#L123) function after creating a [reader object](https://github.com/stfc/fparser/blob/master/src/fparser/api.py#L83) from a source file.

Attached is the test example [test_fparser.zip](https://github.com/stfc/fparser/files/5771679/test_fparser.zip) with the Python test script `test_parsekernel.py` that illustrates the fparser error and uses the attached test kernel `test_fparser_kernel_mod.f90` to trigger the error.

The attached script tries to parse the kernel by two methods:

1. Directly calling the `fparser.api.parse` function as it is done in [`psyclone.parse.kernel.get_kernel_parse_tree`](https://github.com/stfc/PSyclone/blob/master/src/psyclone/parse/kernel.py#L162);
2. Executing step-by-step commands from the `fparser.api.parse` function that calls [`fparser.api.get_reader`](https://github.com/stfc/fparser/blob/master/src/fparser/api.py#L185) and then [`fparser.one.parsefortran.FortranParser.parse`](https://github.com/stfc/fparser/blob/master/src/fparser/api.py#L189) function.

The outcome of these methods depends on the value of `ignore_comments` parameter in the [`fparser.api.get_reader`](https://github.com/stfc/fparser/blob/master/src/fparser/api.py#L84) and [`fparser.api.parse`](https://github.com/stfc/fparser/blob/master/src/fparser/api.py#L151) functions in an fparser installation. Currently both instances are set to `True` in the `fparser.api`script (and hence in an fparser installation) so a kernel parse tree does not include comments. 

- With the default `ignore_comments=True` setting in `get_reader` and `parse` functions the first method, a direct call to `fparser.api.parse` function fails with 

  ```
  File "test_parsekernel.py", line 50, in <module>
    parse_tree = fpapi.parse(filepath)
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/api.py", line 189, in parse
    parser.parse()
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/one/parsefortran.py", line 143, in parse
    raise error
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/one/parsefortran.py", line 128, in parse
    self.block = BeginSource(self)
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/common/base_classes.py", line 776, in __init__
    Statement.__init__(self, parent, item)
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/common/base_classes.py", line 605, in __init__
    self.process_item()
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/one/block_statements.py", line 324, in process_item
    self.fill(end_flag=True)
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/common/base_classes.py", line 827, in fill
    if self.process_subitem(item):
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/one/block_statements.py", line 366, in process_subitem
    return BeginStatement.process_subitem(self, item)
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/common/base_classes.py", line 928, in process_subitem
    self.handle_unknown_item(item)
  File "/data/users/lfric/modules/packages.rhel7/psyclone/1.9.0/lib/python3.6/site-packages/fparser/common/base_classes.py", line 945, in handle_unknown_item
    raise AnalyzeError(message)
  fparser.common.utils.AnalyzeError: While processing 'test_fparser_kernel_mod.f90' (mode='free')..
  219:  end subroutine test_fparser_code
  220:
  221:end module test_fparser_kernel_mod <== no parse pattern found for "end module test_fparser_kernel_mod" in 'BeginSource' block.

  ```

  The second method with the step-by-step commands to create reader and parser objects from `fparser.api.parse` works fine.

- When an fparser installation is hacked to set `ignore_comments` to `False`  in `fparser.api.get_reader` and `fparser.api.parse` functions , the first method, a direct call to `fparser.api.parse` function works fine. The second method with the step-by-step commands to create instances of reader and parser objects also works fine if  `ignore_comments = False` in the attached script. If it is `True`, however, it fails with the same error as the first method with the default `ignore_comments=False`  setting in the `fparser.api` installed module.

When the [maximum depth of the Python interpreter stack](https://docs.python.org/3/library/sys.html#sys.setrecursionlimit) is increased from the default limit of 1000 both the direct call to `fparser.api.parse` and step-by-step approach work regardless of the `ignore_comments` setting in fparser installation and the test script (demonstrated by enbling the commented line `sys.setrecursionlimit(1100)`).

Parsing the offending kernel by fparser2 (`fparser2 test_fparser_kernel_mod.f90`) works fine.

Comments:

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b173dc8..d99544f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,9 @@ Modifications by (in alphabetical order):
 * P. Vitt, University of Siegen, Germany
 * A. Voysey, UK Met Office
 
+18/01/2021 PR #287 for #280. Fixes overly-deep recursion when reading
+           multi-line comments.
+
 11/01/2021 PR #285 for #283. Removes Structure_Constructor_2 class to fix
            problem with Expr matching.
 
diff --git a/src/fparser/common/readfortran.py b/src/fparser/common/readfortran.py
index 60e4ea2..7960f99 100644
--- a/src/fparser/common/readfortran.py
+++ b/src/fparser/common/readfortran.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Modified work Copyright (c) 2017-2020 Science and Technology
+# Modified work Copyright (c) 2017-2021 Science and Technology
 # Facilities Council
 # Original work Copyright (c) 1999-2008 Pearu Peterson
 
@@ -1557,9 +1557,15 @@ class FortranReaderBase(object):
             self.warning(message)
         if name is not None:
             self.error('No construct following construct-name.')
-        if have_comment:
-            return next(self)
-        return self.comment_item('', startlineno, endlineno)
+
+        # If this point is reached, the line is a comment or is
+        # blank. If it is a comment, it has been pushed onto the
+        # fifo_item list.
+        try:
+            return self.fifo_item.pop(0)
+        except IndexError:
+            # A blank line is represented as an empty comment
+            return Comment('', (startlineno, endlineno), self)
 
 
 class FortranFileReader(FortranReaderBase):


Test patch:
diff --git a/src/fparser/common/tests/test_readfortran.py b/src/fparser/common/tests/test_readfortran.py
index 8b75af1..58175e2 100644
--- a/src/fparser/common/tests/test_readfortran.py
+++ b/src/fparser/common/tests/test_readfortran.py
@@ -51,7 +51,7 @@ import pytest
 
 from fparser.common.readfortran import FortranFileReader, \
     FortranStringReader, FortranReaderBase, Line, extract_label, \
-    extract_construct_name, CppDirective
+    extract_construct_name, CppDirective, Comment
 from fparser.common.sourceinfo import FortranFormat
 
 
@@ -1042,7 +1042,7 @@ def test_reader_cpp_directives():
         'end program test']
     ref_text = '\n'.join(input_text[:4] + input_text[5:]).strip()
     reader = FortranStringReader('\n'.join(input_text))
-    lines = [item for item in reader]
+    lines = list(reader)
 
     pp_lines = [1, 3, 5, 6, 8]
     for i, line in enumerate(lines):
@@ -1061,8 +1061,192 @@ def test_multiline_cpp_directives():
         'integer a', 'end program test']
     ref_text = 'program test\n#define ABC   123\ninteger a\nend program test'
     reader = FortranStringReader('\n'.join(input_text))
-    lines = [item for item in reader]
+    lines = list(reader)
     assert len(lines) == 4
     assert isinstance(lines[1], CppDirective)
     assert lines[1].span == (2, 3)
     assert '\n'.join(item.line for item in lines) == ref_text
+
+
+# Tests for the get_source_item method in class FortranReaderBase :
+# Comments
+
+
+def test_single_comment():
+    '''Test that a free format single line comment is output as expected'''
+    input_text = "! a comment\n"
+
+    reader = FortranStringReader(input_text, ignore_comments=False)
+    lines = list(reader)
+    assert len(lines) == 1
+    assert isinstance(lines[0], Comment)
+    assert lines[0].span == (1, 1)
+    assert lines[0].line + "\n" == input_text
+
+    reader = FortranStringReader(input_text, ignore_comments=True)
+    lines = list(reader)
+    assert len(lines) == 0
+
+
+def test_many_comments():
+    '''Test that a large consecutive number of single line comments can be
+    successfully processed. Earlier versions of the reader used
+    recursion for multiple consecutive free format single line
+    comments which resulted in a recursion error in this case.
+
+    '''
+    number_of_comments = 1000
+    input_text = "program hello\n"
+    for index in range(number_of_comments):
+        input_text += "! comment{0}\n".format(index+1)
+    input_text += "end program hello\n"
+
+    reader = FortranStringReader(input_text, ignore_comments=False)
+    lines = list(reader)
+    assert len(lines) == number_of_comments + 2
+    for index in range(1, number_of_comments):
+        assert isinstance(lines[index], Comment)
+        assert lines[index].span == (index+1, index+1)
+        assert lines[index].line + "\n" == "! comment{0}\n".format(index)
+
+    reader = FortranStringReader(input_text, ignore_comments=True)
+    lines = list(reader)
+    assert len(lines) == 2
+
+
+def test_comments_within_continuation():
+    '''Test that comments and multi-line statements are processed
+    correctly.
+
+    '''
+    input_text = (
+        "  ! Comment1\n"
+        "  real :: a &\n"
+        "  ! Comment2\n"
+        "          ,b\n"
+        "  ! Comment3\n")
+
+    reader = FortranStringReader(input_text, ignore_comments=False)
+    lines = list(reader)
+    assert len(lines) == 4
+
+    assert isinstance(lines[0], Comment)
+    assert lines[0].span == (1, 1)
+    assert lines[0].line == "! Comment1"
+
+    assert lines[1].span == (2, 4)
+    assert lines[1].line == "real :: a           ,b"
+
+    assert isinstance(lines[2], Comment)
+    assert lines[2].span == (3, 3)
+    assert lines[2].line == "! Comment2"
+
+    assert isinstance(lines[3], Comment)
+    assert lines[3].span == (5, 5)
+    assert lines[3].line == "! Comment3"
+
+    reader = FortranStringReader(input_text, ignore_comments=True)
+    lines = list(reader)
+    assert len(lines) == 1
+    assert lines[0].span == (2, 4)
+    assert lines[0].line == "real :: a           ,b"
+
+
+# Tests for the get_source_item method in class FortranReaderBase :
+# Blank lines
+
+
+@pytest.mark.parametrize("input_text", ["\n", "    \n"])
+def test_single_blank_line(input_text):
+    '''Test that a single blank line with or without white space is output
+    as an empty Comment object.
+
+    '''
+    reader = FortranStringReader(input_text, ignore_comments=False)
+    lines = list(reader)
+    assert len(lines) == 1
+    assert isinstance(lines[0], Comment)
+    assert lines[0].span == (1, 1)
+    assert lines[0].line == ""
+
+    reader = FortranStringReader(input_text, ignore_comments=True)
+    lines = list(reader)
+    assert len(lines) == 0
+
+
+def test_multiple_blank_lines():
+    '''Test that multiple blank lines with or without white space are
+    output as an empty Comment objects.
+
+    '''
+    input_text = (
+        "   \n\n"
+        "program test\n"
+        "  \n\n"
+        "end program test\n"
+        "  \n\n")
+    reader = FortranStringReader(input_text, ignore_comments=False)
+    lines = list(reader)
+    assert len(lines) == 8
+    for index in [0, 1, 3, 4, 6, 7]:
+        assert isinstance(lines[index], Comment)
+        assert lines[index].span == (index+1, index+1)
+        assert lines[index].line == ""
+    assert isinstance(lines[2], Line)
+    assert lines[2].span == (3, 3)
+    assert lines[2].line == "program test"
+    assert isinstance(lines[5], Line)
+    assert lines[5].span == (6, 6)
+    assert lines[5].line == "end program test"
+
+    reader = FortranStringReader(input_text, ignore_comments=True)
+    lines = list(reader)
+    assert len(lines) == 2
+    assert isinstance(lines[0], Line)
+    assert lines[0].span == (3, 3)
+    assert lines[0].line == "program test"
+    assert isinstance(lines[1], Line)
+    assert lines[1].span == (6, 6)
+    assert lines[1].line == "end program test"
+
+
+def test_blank_lines_within_continuation():
+    '''Test that blank lines and multi-line statements are processed
+    correctly. Note, empty lines within a multi-line statement are
+    removed.
+
+    '''
+    input_text = (
+        "  \n"
+        "  real :: a &\n"
+        "  \n\n"
+        "          ,b\n"
+        "  \n"
+        "  real :: c\n")
+
+    reader = FortranStringReader(input_text, ignore_comments=False)
+    lines = list(reader)
+    assert len(lines) == 4
+
+    assert isinstance(lines[0], Comment)
+    assert lines[0].span == (1, 1)
+    assert lines[0].line == ""
+    assert isinstance(lines[1], Line)
+    assert lines[1].span == (2, 5)
+    assert lines[1].line == "real :: a           ,b"
+    assert isinstance(lines[2], Comment)
+    assert lines[2].span == (6, 6)
+    assert lines[2].line == ""
+    assert isinstance(lines[3], Line)
+    assert lines[3].span == (7, 7)
+    assert lines[3].line == "real :: c"
+
+    reader = FortranStringReader(input_text, ignore_comments=True)
+    lines = list(reader)
+    assert len(lines) == 2
+    assert isinstance(lines[0], Line)
+    assert lines[0].span == (2, 5)
+    assert lines[0].line == "real :: a           ,b"
+    assert isinstance(lines[1], Line)
+    assert lines[1].span == (7, 7)
+    assert lines[1].line == "real :: c"