Repository: eliben/pycparser

Issue:
Parsing '//' comments
I'm getting an error when parsing '//' style comments and not with '/*...*/' style

I've been searching and I've found in pycparser/ply/ctokens.py

```
# Comment (C-Style)
def t_COMMENT(t):
    r'/\*(.|\n)*?\*/'
    t.lexer.lineno += t.value.count('\n')
    return t

# Comment (C++-Style)
def t_CPPCOMMENT(t):
    r'//.*\n'
    t.lexer.lineno += 1
    return t
```

But changing t_COMMENT(t) to the // style doesn't work.

Is there any option to compile with the C++ style? If not, do I need to run yacc or other tool to make it work?

Discussion:
@eliben: The preprocessor is supposed to remove all comments. If you use a preprocessor that supports C++ style comments, this should work because pycparser itself is not aware of comments.@eliben: https://github.com/eliben/pycparser#31interaction-with-the-c-preprocessor@cellularmitosis: @eliben first, thank you so much for your amazing blog and for pycparser.

> The preprocessor is supposed to remove all comments.

I think I found the root cause behind the high volume of these "comments don't work" issues.

It looks like this is a clang vs gcc issue: gcc's cpp removes comments while clang's does not (and clang is the default for macOS users, which is likely where these issue reports are coming from).

```
$ cat foo.c 
int foo() {
    // this is a comment.
    return 42;
}
```

clang:

```
$ /usr/bin/cpp foo.c 
# 1 "foo.c"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 423 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "foo.c" 2
int foo() {
    // this is a comment.
    return 42;
}
```

gcc:

```
$ /opt/homebrew/bin/cpp-14 foo.c 
# 0 "foo.c"
# 0 "<built-in>"
# 0 "<command-line>"
# 1 "foo.c"
int foo() {

    return 42;
}
```

This appears to be a bug in clang, as `clang --help` describes the `-C` option, but there appears to be no inverse of this option:

```
  -C                      Include comments in preprocessed output
```

The easy solution for macOS users appears to be:

```
brew install gcc
```

and then

```
parse_file(filename, use_cpp=True, cpp_path='/opt/homebrew/bin/cpp-14')
```

or

```
preprocess_file(filename, use_cpp=True, cpp_path='/opt/homebrew/bin/cpp-14')
```
@cellularmitosis: Update: I submitted a PR: https://github.com/eliben/pycparser/pull/569

Solution patch:
diff --git a/README.rst b/README.rst
index 388a01e8..ebade8c6 100644
--- a/README.rst
+++ b/README.rst
@@ -110,7 +110,7 @@ Interaction with the C preprocessor
 -----------------------------------
 
 In order to be compilable, C code must be preprocessed by the C preprocessor -
-``cpp``. ``cpp`` handles preprocessing directives like ``#include`` and
+``cpp``. A compatible ``cpp`` handles preprocessing directives like ``#include`` and
 ``#define``, removes comments, and performs other minor tasks that prepare the C
 code for compilation.
 
diff --git a/pycparser/c_lexer.py b/pycparser/c_lexer.py
index 22c64bc7..135826d1 100644
--- a/pycparser/c_lexer.py
+++ b/pycparser/c_lexer.py
@@ -209,6 +209,10 @@ def _make_tok_location(self, token):
 
     bad_octal_constant = '0[0-7]*[89]'
 
+    # comments are not supported
+    unsupported_c_style_comment = r'\/\*'
+    unsupported_cxx_style_comment = r'\/\/'
+
     # character constants (K&R2: A.2.5.2)
     # Note: a-zA-Z and '.-~^_!=&;,' are allowed as escape chars to support #line
     # directives with Windows paths as filenames (..\..\dir\file)
@@ -475,6 +479,16 @@ def t_BAD_CONST_OCT(self, t):
         msg = "Invalid octal constant"
         self._error(msg, t)
 
+    @TOKEN(unsupported_c_style_comment)
+    def t_UNSUPPORTED_C_STYLE_COMMENT(self, t):
+        msg = "Comments are not supported, see https://github.com/eliben/pycparser#3using."
+        self._error(msg, t)
+
+    @TOKEN(unsupported_cxx_style_comment)
+    def t_UNSUPPORTED_CXX_STYLE_COMMENT(self, t):
+        msg = "Comments are not supported, see https://github.com/eliben/pycparser#3using."
+        self._error(msg, t)
+
     @TOKEN(octal_constant)
     def t_INT_CONST_OCT(self, t):
         return t


Test patch:
diff --git a/tests/README.txt b/tests/README.txt
index c540b7b4..6d97ff95 100644
--- a/tests/README.txt
+++ b/tests/README.txt
@@ -1 +1 @@
-Run 'python -m unittest discover' from the root pycparser directory
+Run 'python3 -m unittest discover' from the root repository directory.
diff --git a/tests/test_c_lexer.py b/tests/test_c_lexer.py
index 2975b800..cccb7858 100644
--- a/tests/test_c_lexer.py
+++ b/tests/test_c_lexer.py
@@ -418,11 +418,12 @@ def test_preprocessor_pragma(self):
 ERR_FILENAME_BEFORE_LINE    = 'filename before line'
 ERR_LINENUM_MISSING         = 'line number missing'
 ERR_INVALID_LINE_DIRECTIVE  = 'invalid #line directive'
+ERR_COMMENT                 = 'Comments are not supported'
 
 
 class TestCLexerErrors(unittest.TestCase):
     """ Test lexing of erroneous strings.
-        Works by passing an error functions that saves the error
+        Works by passing an error function that saves the error
         in an attribute for later perusal.
     """
     def error_func(self, msg, line, column):
@@ -496,6 +497,9 @@ def test_preprocessor(self):
         self.assertLexerError('#line "ka"', ERR_FILENAME_BEFORE_LINE)
         self.assertLexerError('#line df', ERR_INVALID_LINE_DIRECTIVE)
         self.assertLexerError('#line \n', ERR_LINENUM_MISSING)
+        # a compatible preprocessor must remove comments.
+        self.assertLexerError('//', ERR_COMMENT)
+        self.assertLexerError('/*', ERR_COMMENT)
 
 
 if __name__ == '__main__':