Repository: tobymao/sqlglot

Issue:
Token col attribute doesn't handle leading whitespace
When tokenizing, the `col` attribute that denotes where the token ends is incorrect if there is a newline followed by leading whitespace.

**Fully reproducible code snippet**
I would expect this behavior: 
```
        tokens = Tokenizer().tokenize("SELECT column\n    FROM table")

        self.assertEqual(tokens[0].line, 1) # SELECT
        self.assertEqual(tokens[0].col, 6)
        self.assertEqual(tokens[1].line, 1) # column
        self.assertEqual(tokens[1].col, 13)
        self.assertEqual(tokens[2].line, 2) # FROM
        self.assertEqual(tokens[2].col, 8)
        self.assertEqual(tokens[3].line, 2) # table
        self.assertEqual(tokens[3].col, 14)
```
but the current implementation would have col==5 for the `FROM` token and col==11 for the `table` token.

**Official Documentation**
[Doc for Token specifies that col is the column where the token ends.](https://sqlglot.com/sqlglot/tokens.html#Token)

Discussion:
@chgiff: I believe the solution is as simple as changing this section in the `_advance` method from a `1` to an `i`.
```
    def _advance(self, i: int = 1, alnum: bool = False) -> None:
        if self.WHITE_SPACE.get(self._char) is TokenType.BREAK:
            # Ensures we don't count an extra line if we get a \r\n line break sequence
            if not (self._char == "\r" and self._peek == "\n"):
                self._col = 1
                self._line += 1
```
should be 
```
...
                self._col = i
...
```
and same for the Rust tokenizer. 

I'm happy to open a PR with the fix if there's agreement this is the correct route to go. @georgesittas: Hey @chgiff, your suggestion seems good. We'll be happy to accept a well-tested PR for this, so feel free to send one our way!

Solution patch:
diff --git a/sqlglot/tokens.py b/sqlglot/tokens.py
index 81d14048fd..c20036eb3a 100644
--- a/sqlglot/tokens.py
+++ b/sqlglot/tokens.py
@@ -1109,7 +1109,7 @@ def _advance(self, i: int = 1, alnum: bool = False) -> None:
         if self.WHITE_SPACE.get(self._char) is TokenType.BREAK:
             # Ensures we don't count an extra line if we get a \r\n line break sequence
             if not (self._char == "\r" and self._peek == "\n"):
-                self._col = 1
+                self._col = i
                 self._line += 1
         else:
             self._col += i
diff --git a/sqlglotrs/src/tokenizer.rs b/sqlglotrs/src/tokenizer.rs
index 461311263d..9849299a10 100644
--- a/sqlglotrs/src/tokenizer.rs
+++ b/sqlglotrs/src/tokenizer.rs
@@ -173,7 +173,7 @@ impl<'a> TokenizerState<'a> {
         if Some(&self.token_types.break_) == self.settings.white_space.get(&self.current_char) {
             // Ensures we don't count an extra line if we get a \r\n line break sequence.
             if !(self.current_char == '\r' && self.peek_char == '\n') {
-                self.column = 1;
+                self.column = i as usize;
                 self.line += 1;
             }
         } else {


Test patch:
diff --git a/tests/test_tokens.py b/tests/test_tokens.py
index 29ef5b61ad..d9242eff28 100644
--- a/tests/test_tokens.py
+++ b/tests/test_tokens.py
@@ -67,9 +67,19 @@ def test_token_line_col(self):
         tokens = Tokenizer().tokenize("SELECT\r\n  1,\r\n  2")
 
         self.assertEqual(tokens[0].line, 1)
+        self.assertEqual(tokens[0].col, 6)
         self.assertEqual(tokens[1].line, 2)
+        self.assertEqual(tokens[1].col, 3)
         self.assertEqual(tokens[2].line, 2)
+        self.assertEqual(tokens[2].col, 4)
         self.assertEqual(tokens[3].line, 3)
+        self.assertEqual(tokens[3].col, 3)
+
+        tokens = Tokenizer().tokenize("  SELECT\n    100")
+        self.assertEqual(tokens[0].line, 1)
+        self.assertEqual(tokens[0].col, 8)
+        self.assertEqual(tokens[1].line, 2)
+        self.assertEqual(tokens[1].col, 7)
 
     def test_crlf(self):
         tokens = Tokenizer().tokenize("SELECT a\r\nFROM b")