Repository: tobymao/sqlglot

Issue:
Windows line endings cause IndexOutOfRange during tokenization
I am parsing view and stored procedure definitions from windows-based mssql.

I am getting failures from IndexOutOfRange and strange parsing contexts where the leading character has been removed.

I think I have identified the problem + solution - are you open to a PR (with unit test)?

The issue is this:

In _advance:
https://github.com/tobymao/sqlglot/blob/a18444dbd7ccfc05b189dcb2005c85a1048cc8de/sqlglot/tokens.py#L1008

If it sees \r\n, then it sets i to 2
_current is increased by i
_char always looks at position _current-1

This can lead to it looking past the end of the source text, as _current-1 > size

I believe the fix is something like changing:
https://github.com/tobymao/sqlglot/blob/a18444dbd7ccfc05b189dcb2005c85a1048cc8de/sqlglot/tokens.py#L1022

to

```
        if self._end:
            self._peek = ""
            self._char = self.sql[self.size - 1]
        else:
            self._char = self.sql[self._current - 1]
            self._peek = self.sql[self._current]
```





Discussion:
Hey @mr-miles, can you share a snippet that reproduces the issue here first?
Good idea!

```
from sqlglot import parse_one, exp

# find all tables (x, y, z)
for table in parse_one("SELECT * FROM x JOIN y JOIN z\r\n").find_all(exp.Table):
    print(table.name)
```

Blows up with qlglot.errors.TokenError

End the query with \r or \n on their own or some other whitespace and it is fine
I see, I can reproduce this issue - thanks! It seems like this is an edge case that we didn't test, apparently! Namely: having the `\r\n` sequence at the end of the SQL.

I'm not sure if your suggestion is the cleanest way to solve this issue, so I will probably look into this as well. Note that changes related to tokenization must be made for both the Python and the Rust [tokenizer](https://github.com/tobymao/sqlglot/blob/main/sqlglotrs/src/tokenizer.rs).

Resolving patch:
diff --git a/sqlglot/tokens.py b/sqlglot/tokens.py
--- a/sqlglot/tokens.py
+++ b/sqlglot/tokens.py
@@ -565,8 +565,7 @@ class Tokenizer(metaclass=_Tokenizer):
         "~": TokenType.TILDA,
         "?": TokenType.PLACEHOLDER,
         "@": TokenType.PARAMETER,
-        # used for breaking a var like x'y' but nothing else
-        # the token type doesn't matter
+        # Used for breaking a var like x'y' but nothing else the token type doesn't matter
         "'": TokenType.QUOTE,
         "`": TokenType.IDENTIFIER,
         '"': TokenType.IDENTIFIER,
@@ -892,7 +891,7 @@ class Tokenizer(metaclass=_Tokenizer):
 
     COMMAND_PREFIX_TOKENS = {TokenType.SEMICOLON, TokenType.BEGIN}
 
-    # handle numeric literals like in hive (3L = BIGINT)
+    # Handle numeric literals like in hive (3L = BIGINT)
     NUMERIC_LITERALS: t.Dict[str, str] = {}
 
     COMMENTS = ["--", ("/*", "*/")]
@@ -965,8 +964,7 @@ def _scan(self, until: t.Optional[t.Callable] = None) -> None:
         while self.size and not self._end:
             current = self._current
 
-            # skip spaces inline rather than iteratively call advance()
-            # for performance reasons
+            # Skip spaces here rather than iteratively calling advance() for performance reasons
             while current < self.size:
                 char = self.sql[current]
 
@@ -975,12 +973,10 @@ def _scan(self, until: t.Optional[t.Callable] = None) -> None:
                 else:
                     break
 
-            n = current - self._current
-            self._start = current
-            self._advance(n if n > 1 else 1)
+            offset = current - self._current if current > self._current else 1
 
-            if self._char is None:
-                break
+            self._start = current
+            self._advance(offset)
 
             if not self._char.isspace():
                 if self._char.isdigit():
@@ -1008,12 +1004,9 @@ def _chars(self, size: int) -> str:
     def _advance(self, i: int = 1, alnum: bool = False) -> None:
         if self.WHITE_SPACE.get(self._char) is TokenType.BREAK:
             # Ensures we don't count an extra line if we get a \r\n line break sequence
-            if self._char == "\r" and self._peek == "\n":
-                i = 2
-                self._start += 1
-
-            self._col = 1
-            self._line += 1
+            if not (self._char == "\r" and self._peek == "\n"):
+                self._col = 1
+                self._line += 1
         else:
             self._col += i
 


Test patch:
diff --git a/tests/test_tokens.py b/tests/test_tokens.py
--- a/tests/test_tokens.py
+++ b/tests/test_tokens.py
@@ -85,6 +85,18 @@ def test_crlf(self):
             ],
         )
 
+        for simple_query in ("SELECT 1\r\n", "\r\nSELECT 1"):
+            tokens = Tokenizer().tokenize(simple_query)
+            tokens = [(token.token_type, token.text) for token in tokens]
+
+            self.assertEqual(
+                tokens,
+                [
+                    (TokenType.SELECT, "SELECT"),
+                    (TokenType.NUMBER, "1"),
+                ],
+            )
+
     def test_command(self):
         tokens = Tokenizer().tokenize("SHOW;")
         self.assertEqual(tokens[0].token_type, TokenType.SHOW)