Repository: tobymao/sqlglot

Issue:
Comment at end of query is removed
Hi,
Could you please help me with this problem?

dialect: trino
version 11.4.5

```
SELECT
  t.name,
  t.tags,
  t.date,
  s.price,
  s.action
FROM "stock prices_grid" AS s
JOIN test_database_1."test data" AS t
  ON s.name = t.name
--test comment will be removed
```

The below code removes the comment at the end which is an unexpected behaviour.
```
expression_tree = parse_one(query, dialect)    
```

Discussion:

Patch:
diff --git a/sqlglot/tokens.py b/sqlglot/tokens.py
index e5b44e7b..a27dfd8a 100644
--- a/sqlglot/tokens.py
+++ b/sqlglot/tokens.py
@@ -946,7 +946,7 @@ class Tokenizer(metaclass=_Tokenizer):
 
         # Leading comment is attached to the succeeding token, whilst trailing comment to the preceding.
         # Multiple consecutive comments are preserved by appending them to the current comments list.
-        if comment_start_line == self._prev_token_line:
+        if comment_start_line == self._prev_token_line or self._end:
             self.tokens[-1].comments.extend(self._comments)
             self._comments = []
             self._prev_token_line = self._line


Test patch:
diff --git a/tests/test_tokens.py b/tests/test_tokens.py
index 909eb18f..8481f4d1 100644
--- a/tests/test_tokens.py
+++ b/tests/test_tokens.py
@@ -13,6 +13,7 @@ class TestTokens(unittest.TestCase):
             ("foo --comment", ["comment"]),
             ("foo", []),
             ("foo /*comment 1*/ /*comment 2*/", ["comment 1", "comment 2"]),
+            ("foo\n-- comment", [" comment"]),
         ]
 
         for sql, comment in sql_comment: