Repository: andialbrecht/sqlparse

Issue:
The group_order() function fails to identify an ordered identifier in the context when nested 
**Describe the bug**

The [`group_order()`](https://github.com/andialbrecht/sqlparse/blob/f101546dafa921edfea5b3107731504665b758ea/sqlparse/engine/grouping.py#L363-L371) function does not leverage the `@recurse` decorator and thus the identifier and `ASC`/`DESC` token aren't grouped together in the context of a subquery.

**To Reproduce**

The following correctly groups `bar DESC` as a single identifier,

```python
>>> from sqlparse import parse
>>>
>>> parse("SELECT * FROM foo ORDER BY bar DESC")[0]._pprint_tree()
|- 0 DML 'SELECT'
|- 1 Whitespace ' '
|- 2 Wildcard '*'
|- 3 Whitespace ' '
|- 4 Keyword 'FROM'
|- 5 Whitespace ' '
|- 6 Identifier 'foo'
|  `- 0 Name 'foo'
|- 7 Whitespace ' '
|- 8 Keyword 'ORDER ...'
|- 9 Whitespace ' '
`- 10 Identifier 'bar DE...'
   |- 0 Identifier 'bar'
   |  `- 0 Name 'bar'
   |- 1 Whitespace ' '
   `- 2 Order 'DESC'
```

whereas when wrapped within a subquery,

```python
>>> from sqlparse import parse

>>> parse("SELECT * FROM (SELECT * FROM foo ORDER BY bar DESC)")[0]._pprint_tree()
|- 0 DML 'SELECT'
|- 1 Whitespace ' '
|- 2 Wildcard '*'
|- 3 Whitespace ' '
|- 4 Keyword 'FROM'
|- 5 Whitespace ' '
`- 6 Parenthesis '(SELEC...'
   |- 0 Punctuation '('
   |- 1 DML 'SELECT'
   |- 2 Whitespace ' '
   |- 3 Wildcard '*'
   |- 4 Whitespace ' '
   |- 5 Keyword 'FROM'
   |- 6 Whitespace ' '
   |- 7 Identifier 'foo'
   |  `- 0 Name 'foo'
   |- 8 Whitespace ' '
   |- 9 Keyword 'ORDER ...'
   |- 10 Whitespace ' '
   |- 11 Identifier 'bar'
   |  `- 0 Name 'bar'
   |- 12 Whitespace ' '
   |- 13 Order 'DESC'
   `- 14 Punctuation ')'
```

the `bar DESC` is not grouped.

**Expected behavior**

The `group_order()` function should leverage the `recurse(sql.Identifier)` decorator to ensure that nested orders are grouped appropriately.

**Versions (please complete the following information):**
 - Python: 3.9
 - sqlparse: 0.4.4

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/andialbrecht/sqlparse/pull/746?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`f101546`)](https://app.codecov.io/gh/andialbrecht/sqlparse/commit/f101546dafa921edfea5b3107731504665b758ea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht) 96.97% compared to head [(`d43fb9c`)](https://app.codecov.io/gh/andialbrecht/sqlparse/pull/746?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht) 96.98%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master     #746   +/-   ##
=======================================
  Coverage   96.97%   96.98%           
=======================================
  Files          20       20           
  Lines        1556     1557    +1     
=======================================
+ Hits         1509     1510    +1     
  Misses         47       47           
```


| [Files](https://app.codecov.io/gh/andialbrecht/sqlparse/pull/746?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht) | Coverage Δ | |
|---|---|---|
| [sqlparse/engine/grouping.py](https://app.codecov.io/gh/andialbrecht/sqlparse/pull/746?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht#diff-c3FscGFyc2UvZW5naW5lL2dyb3VwaW5nLnB5) | `100.00% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/andialbrecht/sqlparse/pull/746?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Andi+Albrecht).


Patch:
diff --git a/AUTHORS b/AUTHORS
index 4617b7d..934bbe3 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -31,9 +31,11 @@ Alphabetical list of contributors:
 * Florian Bauer <florian.bauer@zmdi.com>
 * Fredy Wijaya <fredy.wijaya@gmail.com>
 * Gavin Wahl <gwahl@fusionbox.com>
+* Georg Traar <georg@crate.io>
 * Hugo van Kemenade <hugovk@users.noreply.github.com>
 * hurcy <cinyoung.hur@gmail.com>
 * Ian Robertson <ian.robertson@capitalone.com>
+* Igor Khrol <igor.khrol@automattic.com>
 * JacekPliszka <Jacek.Pliszka@gmail.com>
 * JavierPan <PeterSandwich@users.noreply.github.com>
 * Jean-Martin Archer <jm@jmartin.ca>
diff --git a/CHANGELOG b/CHANGELOG
index 0ede280..cbfbcf2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,10 +10,12 @@ Enhancements:
 
 * Splitting statements now allows to remove the semicolon at the end.
   Some database backends love statements without semicolon (issue742).
+* Support TypedLiterals in get_parameters (pr649, by Khrol).
 
 Bug Fixes
 
 * Ignore dunder attributes when creating Tokens (issue672).
+* Allow operators to precede dollar-quoted strings (issue763).
 
 
 Release 0.4.4 (Apr 18, 2023)
diff --git a/docs/source/extending.rst b/docs/source/extending.rst
index 0c10924..866303b 100644
--- a/docs/source/extending.rst
+++ b/docs/source/extending.rst
@@ -70,7 +70,7 @@ a keyword to the lexer:
     lex.add_keywords(keywords.KEYWORDS)
 
     # add a custom keyword dictionary
-    lex.add_keywords({'BAR', sqlparse.tokens.Keyword})
+    lex.add_keywords({'BAR': sqlparse.tokens.Keyword})
 
     # no configuration is passed here. The lexer is used as a singleton.
     sqlparse.parse("select * from foo zorder by bar;")
diff --git a/sqlparse/engine/grouping.py b/sqlparse/engine/grouping.py
index 57d257e..c486318 100644
--- a/sqlparse/engine/grouping.py
+++ b/sqlparse/engine/grouping.py
@@ -360,6 +360,7 @@ def group_functions(tlist):
         tidx, token = tlist.token_next_by(t=T.Name, idx=tidx)
 
 
+@recurse(sql.Identifier)
 def group_order(tlist):
     """Group together Identifier and Asc/Desc token"""
     tidx, token = tlist.token_next_by(t=T.Keyword.Order)
diff --git a/sqlparse/keywords.py b/sqlparse/keywords.py
index b45f3e0..d3794fd 100644
--- a/sqlparse/keywords.py
+++ b/sqlparse/keywords.py
@@ -30,7 +30,7 @@ SQL_REGEX = [
 
     (r"`(``|[^`])*`", tokens.Name),
     (r"´(´´|[^´])*´", tokens.Name),
-    (r'((?<!\S)\$(?:[_A-ZÀ-Ü]\w*)?\$)[\s\S]*?\1', tokens.Literal),
+    (r'((?<![\w\"\$])\$(?:[_A-ZÀ-Ü]\w*)?\$)[\s\S]*?\1', tokens.Literal),
 
     (r'\?', tokens.Name.Placeholder),
     (r'%(\(\w+\))?s', tokens.Name.Placeholder),
diff --git a/sqlparse/sql.py b/sqlparse/sql.py
index 1ccfbdb..41606dd 100644
--- a/sqlparse/sql.py
+++ b/sqlparse/sql.py
@@ -619,12 +619,14 @@ class Function(NameAliasMixin, TokenList):
     def get_parameters(self):
         """Return a list of parameters."""
         parenthesis = self.tokens[-1]
+        result = []
         for token in parenthesis.tokens:
             if isinstance(token, IdentifierList):
                 return token.get_identifiers()
-            elif imt(token, i=(Function, Identifier), t=T.Literal):
-                return [token, ]
-        return []
+            elif imt(token, i=(Function, Identifier, TypedLiteral),
+                     t=T.Literal):
+                result.append(token)
+        return result
 
 
 class Begin(TokenList):


Test patch:
diff --git a/tests/test_grouping.py b/tests/test_grouping.py
index 03d16c5..e90243b 100644
--- a/tests/test_grouping.py
+++ b/tests/test_grouping.py
@@ -247,6 +247,14 @@ def test_grouping_identifier_list_with_order():
     assert str(p.tokens[0].tokens[3]) == '2 desc'
 
 
+def test_grouping_nested_identifier_with_order():
+    # issue745
+    p = sqlparse.parse('(a desc)')[0]
+    assert isinstance(p.tokens[0], sql.Parenthesis)
+    assert isinstance(p.tokens[0].tokens[1], sql.Identifier)
+    assert str(p.tokens[0].tokens[1]) == 'a desc'
+
+
 def test_grouping_where():
     s = 'select * from foo where bar = 1 order by id desc'
     p = sqlparse.parse(s)[0]
diff --git a/tests/test_parse.py b/tests/test_parse.py
index 5feef5a..be416ef 100644
--- a/tests/test_parse.py
+++ b/tests/test_parse.py
@@ -133,6 +133,11 @@ def test_parse_nested_function():
     assert type(t[0]) is sql.Function
 
 
+def test_parse_casted_params():
+    t = sqlparse.parse("foo(DATE '2023-11-14', TIMESTAMP '2023-11-15')")[0].tokens[0].get_parameters()
+    assert len(t) == 2
+
+
 def test_parse_div_operator():
     p = sqlparse.parse('col1 DIV 5 AS div_col1')[0].tokens
     assert p[0].tokens[0].tokens[2].ttype is T.Operator
@@ -180,6 +185,14 @@ def test_psql_quotation_marks():
     $PROC_2$ LANGUAGE plpgsql;""")
     assert len(t) == 2
 
+    # operators are valid infront of dollar quoted strings
+    t = sqlparse.split("""UPDATE SET foo =$$bar;SELECT bar$$""")
+    assert len(t) == 1
+    
+    # identifiers must be separated by whitespace
+    t = sqlparse.split("""UPDATE SET foo TO$$bar;SELECT bar$$""")
+    assert len(t) == 2
+
 
 def test_double_precision_is_builtin():
     s = 'DOUBLE PRECISION'