nebius/SWE-bench-extra huggingface dataset:
Repository: tobymao/sqlglot

Issue:
List index out of range for Snowflake SQL transpilation
Ran the issue using the latest `main` build `sqlglot-19.0.4.dev18`

Read: `snowflake`
Write `spark` 

When running the following query on Snowflake's publically available documentation, we get an `IndexError: list index out of range`. It's unclear which part of the query is unable to be tokenized. 

Code snippet 

https://docs.snowflake.com/en/sql-reference/sql/create-external-table

```
query_1666 = 'SELECT metadata$filename FROM @s1/;'

transpiled_sqls = transpile(query_1666, read='snowflake', write='spark', pretty=True, error_level=None)
```


Comments:

Patch:
diff --git a/sqlglot/dialects/snowflake.py b/sqlglot/dialects/snowflake.py
index 01f7512a..62b1cd0c 100644
--- a/sqlglot/dialects/snowflake.py
+++ b/sqlglot/dialects/snowflake.py
@@ -349,7 +349,7 @@ class Snowflake(Dialect):
             table: t.Optional[exp.Expression] = None
             if self._match_text_seq("@"):
                 table_name = "@"
-                while True:
+                while self._curr:
                     self._advance()
                     table_name += self._prev.text
                     if not self._match_set(self.STAGED_FILE_SINGLE_TOKENS, advance=False):


Test patch:
diff --git a/tests/dialects/test_snowflake.py b/tests/dialects/test_snowflake.py
index ede42f47..7452a8c9 100644
--- a/tests/dialects/test_snowflake.py
+++ b/tests/dialects/test_snowflake.py
@@ -552,6 +552,7 @@ class TestSnowflake(Validator):
             staged_file.sql(dialect="snowflake"),
         )
 
+        self.validate_identity("SELECT metadata$filename FROM @s1/")
         self.validate_identity("SELECT * FROM @~")
         self.validate_identity("SELECT * FROM @~/some/path/to/file.csv")
         self.validate_identity("SELECT * FROM @mystage")
Repository: tobymao/sqlglot

Issue:
# char at the end of field name disappear when using parse_one with tsql dialect
Query = 'SELECT TestSpecialChar.Test# FROM TestSpecialChar'
Using:
 sql = Parse_one(sql=Query, read='tsql')

Returned value is:
 SELECT TestSpecialChar.Test FROM TestSpecialChar


PS. The query is perfectly valid in T-SQL. I used it to test how SQLGlot was behaving with weird characters in text. Test$ is working, but not Test# 

Comments:

Patch:
diff --git a/sqlglot/dialects/tsql.py b/sqlglot/dialects/tsql.py
index 021843b2..51040ca5 100644
--- a/sqlglot/dialects/tsql.py
+++ b/sqlglot/dialects/tsql.py
@@ -354,6 +354,7 @@ class TSQL(Dialect):
         IDENTIFIERS = ['"', ("[", "]")]
         QUOTES = ["'", '"']
         HEX_STRINGS = [("0x", ""), ("0X", "")]
+        VAR_SINGLE_TOKENS = {"@", "$", "#"}
 
         KEYWORDS = {
             **tokens.Tokenizer.KEYWORDS,


Test patch:
diff --git a/tests/dialects/test_tsql.py b/tests/dialects/test_tsql.py
index 73bcc1ca..d94846cc 100644
--- a/tests/dialects/test_tsql.py
+++ b/tests/dialects/test_tsql.py
@@ -6,6 +6,10 @@ class TestTSQL(Validator):
     dialect = "tsql"
 
     def test_tsql(self):
+        self.validate_identity("SELECT TestSpecialChar.Test# FROM TestSpecialChar")
+        self.validate_identity("SELECT TestSpecialChar.Test@ FROM TestSpecialChar")
+        self.validate_identity("SELECT TestSpecialChar.Test$ FROM TestSpecialChar")
+        self.validate_identity("SELECT TestSpecialChar.Test_ FROM TestSpecialChar")
         self.validate_identity("SELECT TOP (2 + 1) 1")
         self.validate_identity("SELECT * FROM t WHERE NOT c", "SELECT * FROM t WHERE NOT c <> 0")
         self.validate_identity("1 AND true", "1 <> 0 AND (1 = 1)")
Repository: tobymao/sqlglot

Issue:
sqlglot corrupts date_format spec for MySQL
**Before you file an issue**
> - Make sure you specify the "read" dialect eg. parse_one(sql, read="spark")

Yes, `read='mysql'`

> - Check if the issue still exists on main 

Yes

**Fully reproducible code snippet**

> Please include a fully reproducible code snippet or the input sql, dialect, and expected output.

```
In [19]: import sqlglot

In [20]: sqlglot.parse_one("date_format(now(), '%Y-%m-%d %H:%i:00.0000')", read='mysql').sql(dialect='mysql')
Out[20]: "DATE_FORMAT(NOW(), '%Y-%m-%d %H:%M:00.0000')"
```

sqlglot uses `%M` specifier for minute, but in MySQL `%i` should be used.

**Official Documentation**

> Please include links to official SQL documentation related to your issue.

https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_date-format


Comments:

Patch:
diff --git a/sqlglot/time.py b/sqlglot/time.py
index c286ec1e..50ec2ec3 100644
--- a/sqlglot/time.py
+++ b/sqlglot/time.py
@@ -42,6 +42,10 @@ def format_time(
                 end -= 1
                 chars = sym
                 sym = None
+            else:
+                chars = chars[0]
+                end = start + 1
+
             start += len(chars)
             chunks.append(chars)
             current = trie


Test patch:
diff --git a/tests/dialects/test_mysql.py b/tests/dialects/test_mysql.py
index 45bb763b..ab246a3d 100644
--- a/tests/dialects/test_mysql.py
+++ b/tests/dialects/test_mysql.py
@@ -123,6 +123,7 @@ class TestMySQL(Validator):
         self.validate_identity("ALTER TABLE test_table ALTER COLUMN test_column SET DEFAULT 1")
 
     def test_identity(self):
+        self.validate_identity("SELECT DATE_FORMAT(NOW(), '%Y-%m-%d %H:%i:00.0000')")
         self.validate_identity("SELECT @var1 := 1, @var2")
         self.validate_identity("UNLOCK TABLES")
         self.validate_identity("LOCK TABLES `app_fields` WRITE")
Repository: tobymao/sqlglot

Issue:
Inaccurate sql generation for DATEPART in T-SQL
Similar to [2586](https://github.com/tobymao/sqlglot/issues/2586) DATEPART function fails for HOUR and WEEKDAY

**Fully reproducible code snippet**
```
query = """SELECT DATEPART(HOUR, date_and_time) AS 'Hour' FROM table """
print(parse_one(quarter_query, 'tsql').sql(dialect='tsql'))

```
this gives `SELECT FORMAT(CAST(date_and_time AS DATETIME2), 'hour') AS "Hour" FROM table` but the [format code](https://learn.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings) is 'h' . Similarly for WEEKDAY it gives 'dw' which is not present in the format codes(thought I couldnt find a corresponding code)


Comments:

Patch:
diff --git a/sqlglot/dialects/tsql.py b/sqlglot/dialects/tsql.py
index 03f8537b..a60891fe 100644
--- a/sqlglot/dialects/tsql.py
+++ b/sqlglot/dialects/tsql.py
@@ -136,6 +136,9 @@ def _parse_hashbytes(args: t.List) -> exp.Expression:
     return exp.func("HASHBYTES", *args)
 
 
+DATEPART_ONLY_FORMATS = {"dw", "hour", "quarter"}
+
+
 def _format_sql(self: TSQL.Generator, expression: exp.NumberToStr | exp.TimeToStr) -> str:
     fmt = (
         expression.args["format"]
@@ -149,8 +152,8 @@ def _format_sql(self: TSQL.Generator, expression: exp.NumberToStr | exp.TimeToSt
     )
 
     # There is no format for "quarter"
-    if fmt.name.lower() == "quarter":
-        return self.func("DATEPART", "QUARTER", expression.this)
+    if fmt.name.lower() in DATEPART_ONLY_FORMATS:
+        return self.func("DATEPART", fmt.name, expression.this)
 
     return self.func("FORMAT", expression.this, fmt, expression.args.get("culture"))
 


Test patch:
diff --git a/tests/dialects/test_tsql.py b/tests/dialects/test_tsql.py
index 897aa24f..88d1ebcb 100644
--- a/tests/dialects/test_tsql.py
+++ b/tests/dialects/test_tsql.py
@@ -891,8 +891,26 @@ WHERE
         )
 
     def test_datepart(self):
-        self.validate_identity("DATEPART(QUARTER, x)", "DATEPART(QUARTER, CAST(x AS DATETIME2))")
-        self.validate_identity("DATEPART(YEAR, x)", "FORMAT(CAST(x AS DATETIME2), 'yyyy')")
+        self.validate_identity(
+            "DATEPART(QUARTER, x)",
+            "DATEPART(quarter, CAST(x AS DATETIME2))",
+        )
+        self.validate_identity(
+            "DATEPART(YEAR, x)",
+            "FORMAT(CAST(x AS DATETIME2), 'yyyy')",
+        )
+        self.validate_identity(
+            "DATEPART(HOUR, date_and_time)",
+            "DATEPART(hour, CAST(date_and_time AS DATETIME2))",
+        )
+        self.validate_identity(
+            "DATEPART(WEEKDAY, date_and_time)",
+            "DATEPART(dw, CAST(date_and_time AS DATETIME2))",
+        )
+        self.validate_identity(
+            "DATEPART(DW, date_and_time)",
+            "DATEPART(dw, CAST(date_and_time AS DATETIME2))",
+        )
 
         self.validate_all(
             "SELECT DATEPART(month,'1970-01-01')",
Repository: tobymao/sqlglot

Issue:
\r\n sometimes causes a parsing error
Repro

```python
from sqlglot import parse_one
parse_one('select a\r\nfrom b')
``` 

It appears that when the fix went in for line numbers, that the advance function skips the \n, but doesn't advance start, so the text becomes \nfrom which doesn't match.

Comments:

Patch:
diff --git a/sqlglot/tokens.py b/sqlglot/tokens.py
index ef431119..ef3dc237 100644
--- a/sqlglot/tokens.py
+++ b/sqlglot/tokens.py
@@ -587,6 +587,7 @@ class Tokenizer(metaclass=_Tokenizer):
             # Ensures we don't count an extra line if we get a \r\n line break sequence
             if self._char == "\r" and self._peek == "\n":
                 i = 2
+                self._start += 1
 
             self._col = 1
             self._line += 1
diff --git a/sqlglotrs/src/tokenizer.rs b/sqlglotrs/src/tokenizer.rs
index eb800f09..f386ce63 100644
--- a/sqlglotrs/src/tokenizer.rs
+++ b/sqlglotrs/src/tokenizer.rs
@@ -140,6 +140,7 @@ impl<'a> TokenizerState<'a> {
             // Ensures we don't count an extra line if we get a \r\n line break sequence.
             if self.current_char == '\r' && self.peek_char == '\n' {
                 i = 2;
+                self.start += 1;
             }
 
             self.column = 1;


Test patch:
diff --git a/tests/test_tokens.py b/tests/test_tokens.py
index b97f54a6..970c1ac2 100644
--- a/tests/test_tokens.py
+++ b/tests/test_tokens.py
@@ -71,6 +71,20 @@ x"""
         self.assertEqual(tokens[2].line, 2)
         self.assertEqual(tokens[3].line, 3)
 
+    def test_crlf(self):
+        tokens = Tokenizer().tokenize("SELECT a\r\nFROM b")
+        tokens = [(token.token_type, token.text) for token in tokens]
+
+        self.assertEqual(
+            tokens,
+            [
+                (TokenType.SELECT, "SELECT"),
+                (TokenType.VAR, "a"),
+                (TokenType.FROM, "FROM"),
+                (TokenType.VAR, "b"),
+            ],
+        )
+
     def test_command(self):
         tokens = Tokenizer().tokenize("SHOW;")
         self.assertEqual(tokens[0].token_type, TokenType.SHOW)
diff --git a/tests/test_transpile.py b/tests/test_transpile.py
index b732b459..fb8f8313 100644
--- a/tests/test_transpile.py
+++ b/tests/test_transpile.py
@@ -89,6 +89,7 @@ class TestTranspile(unittest.TestCase):
         self.validate("SELECT MIN(3)>=MIN(2)", "SELECT MIN(3) >= MIN(2)")
         self.validate("SELECT 1>0", "SELECT 1 > 0")
         self.validate("SELECT 3>=3", "SELECT 3 >= 3")
+        self.validate("SELECT a\r\nFROM b", "SELECT a FROM b")
 
     def test_comments(self):
         self.validate(
Repository: tobymao/sqlglot

Issue:
ParseError when using LIKE/ILIKE on an element in an object in Snowflake
I'm getting `ParseError: Invalid expression / Unexpected token` when using `LIKE` or `ILIKE` on an element within an object in Snowflake.

Example:
```
import sqlglot
sqlglot.parse(""" select parse_json('{"x": "hello"}'):x like 'hello' """, read="snowflake")
sqlglot.parse(""" select data:x like 'hello' from some_table """, read="snowflake")
```

Both of these cause the parsing error, but both are valid Snowflake statements.

Comments:

Patch:
diff --git a/sqlglot/dialects/snowflake.py b/sqlglot/dialects/snowflake.py
index ad14e6ee..454df94c 100644
--- a/sqlglot/dialects/snowflake.py
+++ b/sqlglot/dialects/snowflake.py
@@ -328,6 +328,9 @@ def _parse_colon_get_path(
         if not self._match(TokenType.COLON):
             break
 
+    if self._match_set(self.RANGE_PARSERS):
+        this = self.RANGE_PARSERS[self._prev.token_type](self, this) or this
+
     return this
 
 


Test patch:
diff --git a/tests/dialects/test_snowflake.py b/tests/dialects/test_snowflake.py
index 602bc630..39963b28 100644
--- a/tests/dialects/test_snowflake.py
+++ b/tests/dialects/test_snowflake.py
@@ -78,6 +78,14 @@ WHERE
         self.validate_identity(
             "SELECT a FROM test PIVOT(SUM(x) FOR y IN ('z', 'q')) AS x TABLESAMPLE (0.1)"
         )
+        self.validate_identity(
+            """SELECT PARSE_JSON('{"x": "hello"}'):x LIKE 'hello'""",
+            """SELECT GET_PATH(PARSE_JSON('{"x": "hello"}'), 'x') LIKE 'hello'""",
+        )
+        self.validate_identity(
+            """SELECT data:x LIKE 'hello' FROM some_table""",
+            """SELECT GET_PATH(data, 'x') LIKE 'hello' FROM some_table""",
+        )
         self.validate_identity(
             "SELECT SUM({ fn CONVERT(123, SQL_DOUBLE) })",
             "SELECT SUM(CAST(123 AS DOUBLE))",
Repository: tobymao/sqlglot

Issue:
sqlglot 20.8.0 incorrectly transpiles or fails to parse Snowflake REMOVE / RM syntax
Snowflake RM / REMOVE syntax is incorrectly transpiled or causes errors for the cases from the documentation: `"RM @parquet_stage;" -> "RM AS $parquet_stage;"` is rejected by Snowflake at runtime, the other cases cause ParseErrors.

## reference
https://docs.snowflake.com/en/sql-reference/sql/remove

## code
```
examples = [
    "RM @parquet_stage;",
    "REMOVE @parquet_stage;",
    "RM @%mytable/myobject;",
    "RM @%mytable/myobject/;",
    "RM @~ pattern='.*jun.*';",
    "REMOVE @%orders;",
    "REMOVE @mystage/path1/subpath2;",
]

import sqlglot
import sqlglot.errors

print(sqlglot.__version__)
for i, s in enumerate(examples):
    try:
        t = sqlglot.transpile(
            s.strip(), read="snowflake", write="snowflake", pretty=True
        )
        print(i, s, "->", t)
    except sqlglot.errors.ParseError as e:
        print(i, "error", e)
```
## output
```
20.8.0
0 RM @parquet_stage; -> ['RM AS $parquet_stage']
1 REMOVE @parquet_stage; -> ['REMOVE AS $parquet_stage']
2 error Required keyword: 'this' missing for <class 'sqlglot.expressions.Parameter'>. Line 1, Col: 5.
  RM @%mytable/myobject;
3 error Required keyword: 'this' missing for <class 'sqlglot.expressions.Parameter'>. Line 1, Col: 5.
  RM @%mytable/myobject/;
4 error Required keyword: 'this' missing for <class 'sqlglot.expressions.Parameter'>. Line 1, Col: 5.
  RM @~ pattern='.*jun.*';
5 error Required keyword: 'this' missing for <class 'sqlglot.expressions.Parameter'>. Line 1, Col: 9.
  REMOVE @%orders;
6 error Invalid expression / Unexpected token. Line 1, Col: 16.
  REMOVE @mystage/path1/subpath2;
```

Comments:

Patch:
diff --git a/sqlglot/dialects/snowflake.py b/sqlglot/dialects/snowflake.py
index f88f3eb1..6ca8e9de 100644
--- a/sqlglot/dialects/snowflake.py
+++ b/sqlglot/dialects/snowflake.py
@@ -663,7 +663,9 @@ class Snowflake(Dialect):
             "MINUS": TokenType.EXCEPT,
             "NCHAR VARYING": TokenType.VARCHAR,
             "PUT": TokenType.COMMAND,
+            "REMOVE": TokenType.COMMAND,
             "RENAME": TokenType.REPLACE,
+            "RM": TokenType.COMMAND,
             "SAMPLE": TokenType.TABLE_SAMPLE,
             "SQL_DOUBLE": TokenType.DOUBLE,
             "SQL_VARCHAR": TokenType.VARCHAR,


Test patch:
diff --git a/tests/dialects/test_snowflake.py b/tests/dialects/test_snowflake.py
index 1f365bea..5dd81cdc 100644
--- a/tests/dialects/test_snowflake.py
+++ b/tests/dialects/test_snowflake.py
@@ -39,6 +39,8 @@ WHERE
   )""",
         )
 
+        self.validate_identity("RM @parquet_stage")
+        self.validate_identity("REMOVE @parquet_stage")
         self.validate_identity("SELECT TIMESTAMP_FROM_PARTS(d, t)")
         self.validate_identity("SELECT GET_PATH(v, 'attr[0].name') FROM vartab")
         self.validate_identity("SELECT TO_ARRAY(CAST(x AS ARRAY))")
Repository: tobymao/sqlglot

Issue:
Support User Defined Functions on Athena Dialect
It looks like sqlglot is not able to parse [AWS Athena's user defined functions syntax](https://docs.aws.amazon.com/athena/latest/ug/querying-udf.html):

```py
from sqlglot import parse
from sqlglot.dialects import Trino

parse("""
    USING EXTERNAL FUNCTION some_function(input VARBINARY)
        RETURNS VARCHAR
        LAMBDA 'some-name'
    SELECT
        some_function(1)
""", dialect=Trino)
```

Exception:

```
sqlglot.errors.ParseError: Invalid expression / Unexpected token. Line 2, Col: 9.
  
    USING EXTERNAL FUNCTION some_function(input VARBINARY)
        RETURNS VARCHAR
        LAMBDA 'some-name'
```

We are using `Trino` dialect since sqlglot does not have a dedicated one for Athena, as far as I understand, but Athena is based off Trino, so this dialect works otherwise perfectly for our codebase :slightly_smiling_face: 

Am I missing something? Does it need a dedicated dialect for Athena?


Comments:

Patch:
diff --git a/sqlglot/dialects/__init__.py b/sqlglot/dialects/__init__.py
index 276ad59c..29c65800 100644
--- a/sqlglot/dialects/__init__.py
+++ b/sqlglot/dialects/__init__.py
@@ -61,6 +61,7 @@ dialect implementations in order to understand how their various components can
 ----
 """
 
+from sqlglot.dialects.athena import Athena
 from sqlglot.dialects.bigquery import BigQuery
 from sqlglot.dialects.clickhouse import ClickHouse
 from sqlglot.dialects.databricks import Databricks
diff --git a/sqlglot/dialects/athena.py b/sqlglot/dialects/athena.py
new file mode 100644
index 00000000..dc87d8dc
--- /dev/null
+++ b/sqlglot/dialects/athena.py
@@ -0,0 +1,12 @@
+from __future__ import annotations
+
+from sqlglot.dialects.trino import Trino
+from sqlglot.tokens import TokenType
+
+
+class Athena(Trino):
+    class Parser(Trino.Parser):
+        STATEMENT_PARSERS = {
+            **Trino.Parser.STATEMENT_PARSERS,
+            TokenType.USING: lambda self: self._parse_as_command(self._prev),
+        }
diff --git a/sqlglot/dialects/dialect.py b/sqlglot/dialects/dialect.py
index f11c0da2..d2533ebc 100644
--- a/sqlglot/dialects/dialect.py
+++ b/sqlglot/dialects/dialect.py
@@ -31,6 +31,7 @@ class Dialects(str, Enum):
 
     DIALECT = ""
 
+    ATHENA = "athena"
     BIGQUERY = "bigquery"
     CLICKHOUSE = "clickhouse"
     DATABRICKS = "databricks"


Test patch:
diff --git a/tests/dialects/test_athena.py b/tests/dialects/test_athena.py
new file mode 100644
index 00000000..99e36f21
--- /dev/null
+++ b/tests/dialects/test_athena.py
@@ -0,0 +1,16 @@
+from tests.dialects.test_dialect import Validator
+
+
+class TestAthena(Validator):
+    dialect = "athena"
+    maxDiff = None
+
+    def test_athena(self):
+        self.validate_identity(
+            """USING EXTERNAL FUNCTION some_function(input VARBINARY)
+            RETURNS VARCHAR
+                LAMBDA 'some-name'
+            SELECT
+            some_function(1)""",
+            check_command_warning=True,
+        )
Repository: tobymao/sqlglot

Issue:
running simplify multiple times fails
**Fully reproducible code snippet**
Please include a fully reproducible code snippet or the input sql, dialect, and expected output.

```
>>> from sqlglot.optimizer.scope import exp
>>> from sqlglot.optimizer.simplify import simplify                                                                                 
>>> from sqlglot import parse_one
>>> ast=parse_one("SELECT a,c,b from table1 where 1 = 1;")
>>> w = ast.find(exp.Where)
>>> w2 = simplify(w)
>>> w2 = simplify(w)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib/python3.9/site-packages/sqlglot/optimizer/simplify.py", line 109, in simplify
    expression = while_changing(expression, _simplify)
  File "lib/python3.9/site-packages/sqlglot/helper.py", line 211, in while_changing
    expression = func(expression)
  File "lib/python3.9/site-packages/sqlglot/optimizer/simplify.py", line 106, in _simplify
    expression.replace(node)
  File "lib/python3.9/site-packages/sqlglot/expressions.py", line 624, in replace
    value = parent.args[key]
KeyError: 'where'
>>> 
```

in my code i run simplify multiple times as I reduce some elements of a query. in this case, elements in the where clause.

as you see in the sample code, first time works fine but then the where clause was removed and the code fails. 
This used to work before, so I guess some check that they key exists has been removed.


Comments:

Patch:
diff --git a/sqlglot/expressions.py b/sqlglot/expressions.py
index 7a8fef4e..21269329 100644
--- a/sqlglot/expressions.py
+++ b/sqlglot/expressions.py
@@ -621,7 +621,7 @@ class Expression(metaclass=_Expression):
             return expression
 
         key = self.arg_key
-        value = parent.args[key]
+        value = parent.args.get(key)
 
         if isinstance(value, list):
             index = self.index
@@ -639,7 +639,7 @@ class Expression(metaclass=_Expression):
                 else:
                     value[index] = expression
                     parent._set_parent(key, expression, index=index)
-        else:
+        elif value is not None:
             if expression is None:
                 parent.args.pop(key)
             else:


Test patch:
diff --git a/tests/test_optimizer.py b/tests/test_optimizer.py
index 71c65fae..984ec22b 100644
--- a/tests/test_optimizer.py
+++ b/tests/test_optimizer.py
@@ -342,6 +342,9 @@ class TestOptimizer(unittest.TestCase):
     def test_simplify(self):
         self.check_file("simplify", simplify, set_dialect=True)
 
+        expression = parse_one("SELECT a, c, b FROM table1 WHERE 1 = 1")
+        self.assertEqual(simplify(simplify(expression.find(exp.Where))).sql(), "WHERE TRUE")
+
         expression = parse_one("TRUE AND TRUE AND TRUE")
         self.assertEqual(exp.true(), optimizer.simplify.simplify(expression))
         self.assertEqual(exp.true(), optimizer.simplify.simplify(expression.this))