Repository: andialbrecht/sqlparse

Issue:
Statement.get_type() does not skip comments between subqueries.
SQL query can contain comment in between WITH multiple query but the `get_type()` implementation doesn't skip them

```python
>>> query, = sqlparse.parse("""
WITH A AS (),
          -- A comment about the B subquery...
          B AS ()
SELECT * FROM B
""")
>>> query.get_type()
'UNKNOWN'
```

Without the comment:

```python
>>> query, = sqlparse.parse("""
WITH A AS (),
          B AS ()
SELECT * FROM B
""")
>>> query.get_type()
'SELECT'
```

Discussion:
mrmasterplan: I like this change. This should be merged. You also added tests, thanks for that. Your test passes when I run it locally. Thanks for your contribution.

It seems that the tests did not run in the PR pipeline. I don't see why they didn't. 
@andialbrecht, do you have any thoughts on this?
shikanime: There is apparently a better solution by fixing the parser itself but so far I haven't found the main source of the problem.
It seems that the `<Comment>.ttype` is empty for some reason during the grouping process to `IdentifierList` (sqlparse/engine/filter_stack.py:36).
Therefore, the subsequent `get_type()` command has no appropriate AST to interpret, but for some statements such as Select are working for maybe some other reason downstream.
andialbrecht: It seems that the github actions were configured after this pr was created. Before github actions the tests ran on Travis CI. @shikanime maybe you can rebase your changes to the current sources. That should trigger the test pipeline.

Patch:
diff --git a/sqlparse/sql.py b/sqlparse/sql.py
index 586cd21..1ccfbdb 100644
--- a/sqlparse/sql.py
+++ b/sqlparse/sql.py
@@ -413,27 +413,28 @@ class Statement(TokenList):
         Whitespaces and comments at the beginning of the statement
         are ignored.
         """
-        first_token = self.token_first(skip_cm=True)
-        if first_token is None:
+        token = self.token_first(skip_cm=True)
+        if token is None:
             # An "empty" statement that either has not tokens at all
             # or only whitespace tokens.
             return 'UNKNOWN'
 
-        elif first_token.ttype in (T.Keyword.DML, T.Keyword.DDL):
-            return first_token.normalized
+        elif token.ttype in (T.Keyword.DML, T.Keyword.DDL):
+            return token.normalized
 
-        elif first_token.ttype == T.Keyword.CTE:
+        elif token.ttype == T.Keyword.CTE:
             # The WITH keyword should be followed by either an Identifier or
             # an IdentifierList containing the CTE definitions;  the actual
             # DML keyword (e.g. SELECT, INSERT) will follow next.
-            fidx = self.token_index(first_token)
-            tidx, token = self.token_next(fidx, skip_ws=True)
-            if isinstance(token, (Identifier, IdentifierList)):
-                _, dml_keyword = self.token_next(tidx, skip_ws=True)
-
-                if dml_keyword is not None \
-                        and dml_keyword.ttype == T.Keyword.DML:
-                    return dml_keyword.normalized
+            tidx = self.token_index(token)
+            while tidx is not None:
+                tidx, token = self.token_next(tidx, skip_ws=True)
+                if isinstance(token, (Identifier, IdentifierList)):
+                    tidx, token = self.token_next(tidx, skip_ws=True)
+
+                    if token is not None \
+                            and token.ttype == T.Keyword.DML:
+                        return token.normalized
 
         # Hmm, probably invalid syntax, so return unknown.
         return 'UNKNOWN'


Test patch:
diff --git a/tests/test_regressions.py b/tests/test_regressions.py
index 4ffc69f..bc8b7dd 100644
--- a/tests/test_regressions.py
+++ b/tests/test_regressions.py
@@ -427,3 +427,12 @@ def test_splitting_at_and_backticks_issue588():
         'grant foo to user1@`myhost`; grant bar to user1@`myhost`;')
     assert len(splitted) == 2
     assert splitted[-1] == 'grant bar to user1@`myhost`;'
+
+
+def test_comment_between_cte_clauses_issue632():
+    p, = sqlparse.parse("""
+        WITH foo AS (),
+             -- A comment before baz subquery
+             baz AS ()
+        SELECT * FROM baz;""")
+    assert p.get_type() == "SELECT"