Repository: coala/coala

Issue:
DocumentationAPI Bug with `C` test cases.
We now use `markers.split()` to find docstrings. Amend the changes to `assemble()`
https://github.com/coala/coala/pull/4530/files#r128902524
Opened by @damngamerz at [gitter//damngamerz](https://gitter.im//damngamerz?at=597591db614889d475e41f4e)


Discussion:
Tried amending the changes to `assemble()` This breaks all other tests. Not able to find a proper solution
we use `marker.split()` to find docstrings...
This would successfully parse Case 1 and 2.
Case 1
```
/**
 * Returns an String that says Hello with the name argument.
 *
 * @param  name the name to which to say hello
 * @return      the concatenated string
 */
```
Case 2
```
/**
* Returns an String that says Hello with the name argument.
*
* @param  name the name to which to say hello
* @return      the concatenated string
*/
```
But when we apply `marker.split()` in `assemble()` (which is a probable solution to this bug). The Case 1 docstring would be assembled to Case 2. Which is the intended behavior but will break whole API.
Case 3:
```
/**
* Returns an String that says Hello with the name argument.
*
* @param  name the name to which to say hello
* @return      the concatenated string*/
```
markers `['/**',' *',' */']`
we can't fix this by expanding `indent`( with markers), as at some point I have to integrate it into the `assemble()`. Which will break all other test cases as `assemble()` is the only thing which makes sure that the markers are perfect(as markers amending is done in `assemble`). And by expanding the `indent`(with markers), we will be introducing faulty markers into `assemble()`.

Right now what I think DocumentationAPI doesn't care about something which has faulty markers. So best solution is to raise error.
I'm not sure I understand. 

Your idea was to have an additional set that defines the indentation of each marker: `marker_indentation = [1, 2, 1]` for example, right?

If you read them during extraction and apply them again during `assemble` what would go wrong?
Indeed, this was the solution we discussed.
Now what will probably go wrong is....
We assemble markers at the end i.e. in `assemble()`. Which takes care of perfect markers being assembled. Now by applying the solution we will be introducing faulty markers. This will fix the corner case, but the whole picture will get disturbed. And we wont be able to fix those faulty markers(as its been amended at last by `assemble`). 
The thing I meant is that DocumentationAPI will not care about anything if it's markers are faulty. From the extraction till the assembling if something is wrong with the markers itself, documentationAPI will just ignore them and fails.

Resolving patch:
diff --git a/coalib/bearlib/languages/documentation/DocBaseClass.py b/coalib/bearlib/languages/documentation/DocBaseClass.py
--- a/coalib/bearlib/languages/documentation/DocBaseClass.py
+++ b/coalib/bearlib/languages/documentation/DocBaseClass.py
@@ -39,8 +39,9 @@ def extract(content, language, docstyle):
                                    defined in given docstyle.
         :raises ValueError:        Raised when a docstyle definition setting
                                    has an invalid format.
-        :return:                   An iterator returning each
-                                   DocumentationComment found in the content.
+        :return:                   An iterator returning instances of
+                                   DocumentationComment or MalformedComment
+                                   found in the content.
         """
         docstyle_definition = DocstyleDefinition.load(language, docstyle)
         return extract_documentation_with_markers(
diff --git a/coalib/bearlib/languages/documentation/DocumentationComment.py b/coalib/bearlib/languages/documentation/DocumentationComment.py
--- a/coalib/bearlib/languages/documentation/DocumentationComment.py
+++ b/coalib/bearlib/languages/documentation/DocumentationComment.py
@@ -284,3 +284,29 @@ def assemble(self):
         assembled = ('\n' * self.top_padding + assembled +
                      '\n' * self.bottom_padding)
         return assembled
+
+
+class MalformedComment:
+    """
+    The MalformedComment holds information about the errors generated by the
+    DocumentationExtraction, DocumentationComment, DocstyleDefinition and
+    DocBaseClass.
+
+    When these classes are unable to parse certain docstrings, an instance
+    of MalformedComment will be returned instead of DocumentationComment.
+    """
+
+    def __init__(self, message, line):
+        """
+        Instantiate a MalformedComment, which contains the information about
+        the error: a message explaining the behaviour and a line no where the
+        error has occured.
+
+        :param message:
+            Contains the message about the error.
+        :param line:
+            Contains the current line of the docstring where the error has
+            occured.
+        """
+        self.message = message
+        self.line = line
diff --git a/coalib/bearlib/languages/documentation/DocumentationExtraction.py b/coalib/bearlib/languages/documentation/DocumentationExtraction.py
--- a/coalib/bearlib/languages/documentation/DocumentationExtraction.py
+++ b/coalib/bearlib/languages/documentation/DocumentationExtraction.py
@@ -1,8 +1,9 @@
 import re
 
 from coalib.bearlib.languages.documentation.DocumentationComment import (
-    DocumentationComment)
+    DocumentationComment, MalformedComment)
 from coalib.results.TextPosition import TextPosition
+from textwrap import dedent
 
 
 def _extract_doc_comment_simple(content, line, column, markers):
@@ -123,22 +124,21 @@ def _extract_doc_comment_standard(content, line, column, markers):
     line += 1
 
     while line < len(content):
-        pos = content[line].find(markers[2].strip())
-        each_line_pos = content[line].find(markers[1].strip())
+        pos = content[line].find(markers[2])
+        each_line_pos = content[line].find(markers[1])
 
         if pos == -1:
             if each_line_pos == -1:
                 # If the first text occurrence is not the each-line marker
                 # now we violate the doc-comment layout.
                 return None
-            doc_comment += content[line][
-                           each_line_pos + len(markers[1].strip()):]
+            doc_comment += content[line][each_line_pos + len(markers[1]):]
         else:
             # If no each-line marker found or it's located past the end marker:
             # extract no further and end the doc-comment.
             if each_line_pos != -1 and each_line_pos + 1 < pos:
                 doc_comment += content[line][each_line_pos +
-                                             len(markers[1].strip()):pos]
+                                             len(markers[1]):pos]
 
             return line, pos + len(markers[2]), doc_comment
 
@@ -197,7 +197,16 @@ def _extract_doc_comment_from_line(content, line, column, regex,
                 doc = DocumentationComment(documentation, docstyle_definition,
                                            indent, marker, position)
 
-                return end_line, end_column, doc
+                break
+
+        if doc_comment:
+            return end_line, end_column, doc
+        else:
+            malformed_comment = MalformedComment(dedent("""\
+                Please check the docstring for faulty markers. A starting
+                marker has been found, but no instance of DocComment is
+                returned."""), line)
+            return line + 1, 0, malformed_comment
 
     return line + 1, 0, None
 
@@ -245,7 +254,10 @@ def extract_documentation_with_markers(content, docstyle_definition):
             begin_regex,
             marker_dict,
             docstyle_definition)
-        if doc:
+
+        if doc and isinstance(doc, MalformedComment):
+            yield doc
+        elif doc:
             # Ignore string literals
             ignore_regex = re.compile(
                 '^\s*r?(?P<marker>' +


Test patch:
diff --git a/tests/bearlib/languages/documentation/DocBaseClassTest.py b/tests/bearlib/languages/documentation/DocBaseClassTest.py
--- a/tests/bearlib/languages/documentation/DocBaseClassTest.py
+++ b/tests/bearlib/languages/documentation/DocBaseClassTest.py
@@ -3,7 +3,7 @@
 from coalib.bearlib.languages.documentation.DocstyleDefinition import (
     DocstyleDefinition)
 from coalib.bearlib.languages.documentation.DocumentationComment import (
-    DocumentationComment)
+    DocumentationComment, MalformedComment)
 from coalib.bearlib.languages.documentation.DocBaseClass import (
     DocBaseClass)
 from tests.bearlib.languages.documentation.TestUtils import (
@@ -11,6 +11,7 @@
 from coalib.results.TextPosition import TextPosition
 from coalib.results.TextRange import TextRange
 from coalib.results.Diff import Diff
+from textwrap import dedent
 
 
 class DocBaseClassTest(unittest.TestCase):
@@ -72,7 +73,7 @@ def test_DocBaseClass_extraction_C_2(self):
 
         self.assertEqual(
             list(DocBaseClass.extract(data, 'C', 'doxygen')),
-            [DocumentationComment(' my main description\n continues here ',
+            [DocumentationComment(' my main description\n continues here',
                                   docstyle_C_doxygen, '',
                                   docstyle_C_doxygen.markers[0],
                                   TextPosition(1, 1))])
@@ -301,3 +302,47 @@ def test_DocBaseClass_process_documentation_not_implemented(self):
         test_object = DocBaseClass()
         self.assertRaises(NotImplementedError,
                           test_object.process_documentation)
+
+    def test_MalformedComment1_C(self):
+        data = ['/**\n',
+                '* A doc-comment aborted in the middle of writing\n',
+                '* This won\'t get parsed (hopefully...)\n']
+
+        expected = [dedent("""\
+            Please check the docstring for faulty markers. A starting
+            marker has been found, but no instance of DocComment is
+            returned."""), 0]
+
+        for doc_comment in DocBaseClass.extract(data, 'C', 'doxygen'):
+            self.assertEqual(
+                [doc_comment.message, doc_comment.line],
+                expected)
+
+    def test_MalformedComment2_CPP(self):
+        data = ['\n',
+                '/** Aborts...\n']
+
+        expected = [dedent("""\
+            Please check the docstring for faulty markers. A starting
+            marker has been found, but no instance of DocComment is
+            returned."""), 1]
+
+        for doc_comment in DocBaseClass.extract(data, 'CPP', 'doxygen'):
+            self.assertEqual(
+                [doc_comment.message, doc_comment.line],
+                expected)
+
+    def test_MalformedComment3_JAVA(self):
+        data = ['/**\n',
+                '* Markers are faulty\n',
+                '*/']
+
+        expected = [dedent("""\
+            Please check the docstring for faulty markers. A starting
+            marker has been found, but no instance of DocComment is
+            returned."""), 0]
+
+        for doc_comment in DocBaseClass.extract(data, 'JAVA', 'default'):
+            self.assertEqual(
+                [doc_comment.message, doc_comment.line],
+                expected)
diff --git a/tests/bearlib/languages/documentation/DocumentationCommentTest.py b/tests/bearlib/languages/documentation/DocumentationCommentTest.py
--- a/tests/bearlib/languages/documentation/DocumentationCommentTest.py
+++ b/tests/bearlib/languages/documentation/DocumentationCommentTest.py
@@ -243,12 +243,7 @@ def test_java_default(self):
                      self.ExceptionValue(name='IOException',
                                          desc='throws IOException\n'),
                      self.ReturnValue(
-                         desc='     the concatenated string\n')],
-                    [self.Description(
-                     desc='\n Returns Area of a square.\n\n'),
-                     self.Parameter(name='side', desc='side of square\n'),
-                     self.ReturnValue(desc=' area of a square\n')],
-                    ]
+                         desc='     the concatenated string\n')]]
 
         self.assertEqual(expected, parsed_docs)
 
diff --git a/tests/bearlib/languages/documentation/documentation_extraction_testdata/data.c b/tests/bearlib/languages/documentation/documentation_extraction_testdata/data.c
--- a/tests/bearlib/languages/documentation/documentation_extraction_testdata/data.c
+++ b/tests/bearlib/languages/documentation/documentation_extraction_testdata/data.c
@@ -31,8 +31,3 @@ int myfield;
 int foobar(int x) {
     return x * x - x + 1;
 }
-
-
-/**
- * A doc-comment aborted in the middle of writing
- * This won't get parsed (hopefully...)
diff --git a/tests/bearlib/languages/documentation/documentation_extraction_testdata/data2.cpp b/tests/bearlib/languages/documentation/documentation_extraction_testdata/data2.cpp
--- a/tests/bearlib/languages/documentation/documentation_extraction_testdata/data2.cpp
+++ b/tests/bearlib/languages/documentation/documentation_extraction_testdata/data2.cpp
@@ -1,5 +1,3 @@
 /**module comment
  * hello world
  */
-
-/** Aborts...
diff --git a/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.c b/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.c
--- a/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.c
+++ b/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.c
@@ -33,11 +33,6 @@ int foobar(int x) {
 }
 
      /** line 1
-      *  line2
-      */
+      *  line2 */
 
     /** */
-
-/**
- * A doc-comment aborted in the middle of writing
- * This won't get parsed (hopefully...)
diff --git a/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.java b/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.java
--- a/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.java
+++ b/tests/bearlib/languages/documentation/documentation_extraction_testdata/default.java
@@ -11,16 +11,3 @@ public String sayHello(String name) throws IOException {
         return "Hello, " + name;
     }
 }
-
-class AreaSquare {
-
-/**
-* Returns Area of a square.
-*
-* @param  side side of square
-* @return  area of a square
-*/
-    public String sayHello(int side) {
-        return side * side;
-    }
-}