Repository: sqlfluff/sqlfluff

Issue:
Commented dash character converted to non utf-8 character
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### What Happened

Upon fixing a query containing a multi-line comment, SQLFluff attempts to fix a commented line.

This:
```sql
/*
TODO
 - tariff scenario —> dm_tariff_scenario
*/
```

Became:
```sql
/*
TODO
 - tariff scenario > dm_tariff_scenario
*/
``` 
This in an invisible char represented as `<97>`

This causes an issue with dbt which can not compile with this char present

Note this comment comes at the end of the file.

### Expected Behaviour

Does not replace/fix anything that is commented

### Observed Behaviour

```bash
 $  sqlfluff fix dbt/models/marts/core/f_utility_statements.sql                                                                                                                                                                                               
==== finding fixable violations ====                                                                                                                                                                                                                          
=== [dbt templater] Sorting Nodes...                                                                                                                                                                                                                          
=== [dbt templater] Compiling dbt project...                                                                                                                                                                                                                  
=== [dbt templater] Project Compiled.                                                                                                                                                                                                                         
== [dbt/models/marts/core/f_utility_statements.sql] FAIL                                                                                                                                                                                                      
L:   1 | P:   5 | L001 | Unnecessary trailing whitespace.                                                                                                                                                                                                     
L:   2 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]                                                                                                                                                                               
L:   3 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]                                                                                                                                                                               
L:   4 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]                                                                                                                                                                               
L:   4 | P:   6 | L019 | Found trailing comma. Expected only leading.                                                                                                                                                                                         
L:   6 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]                                                                                                                                                                               
L:   7 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]                                                                                                                                                                               
L:   8 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]                                                                                                                                                                               
L:   8 | P:   6 | L019 | Found trailing comma. Expected only leading.                                                                                                                                                                                         
L:  10 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]                                                                                                                                                                               
L:  11 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]                                                                                                                                                                               
L:  12 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]                                                                                                                                                                               
L:  12 | P:   6 | L019 | Found trailing comma. Expected only leading.                                                                                                                                                                                         
L:  15 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]   
L:  16 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]                                                                                                                                                                      [0/47960]
L:  17 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  18 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  19 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  20 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  20 | P:  36 | L031 | Avoid aliases in from clauses and join conditions.
L:  21 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  21 | P:  32 | L031 | Avoid aliases in from clauses and join conditions.
L:  22 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]
L:  22 | P:   6 | L019 | Found trailing comma. Expected only leading.
L:  24 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]
L:  26 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  26 | P:  15 | L001 | Unnecessary trailing whitespace.
L:  27 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  28 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  29 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  30 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  31 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  32 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  32 | P:  24 | L011 | Implicit/explicit aliasing of table.
L:  32 | P:  24 | L031 | Avoid aliases in from clauses and join conditions.
L:  33 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  33 | P:  49 | L011 | Implicit/explicit aliasing of table.
L:  33 | P:  49 | L031 | Avoid aliases in from clauses and join conditions.
L:  33 | P:  52 | L001 | Unnecessary trailing whitespace.
L:  34 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  36 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  37 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]
L:  37 | P:   6 | L019 | Found trailing comma. Expected only leading.
L:  39 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]
L:  41 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  41 | P:   9 | L034 | Select wildcards then simple targets before calculations
                       | and aggregates.
L:  43 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  46 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  47 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  48 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  51 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  52 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  53 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  54 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  57 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  58 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  61 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  62 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  64 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  65 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  68 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  69 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  70 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  71 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  73 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  73 | P:  36 | L031 | Avoid aliases in from clauses and join conditions.
L:  74 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  74 | P:  56 | L031 | Avoid aliases in from clauses and join conditions.
L:  75 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  76 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  76 | P:  28 | L001 | Unnecessary trailing whitespace.
L:  77 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  80 | P:   9 | L003 | Expected 0 indentations, found 2 [compared to line 01]
L:  81 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  83 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  84 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]
L:  94 | P:   1 | L009 | Files must end with a single trailing newline.
```

### How to reproduce

`sqlfluff fix` with provided `.sqlfluff` configuration

SQL contains proprietary code and I am, likely, unable to provide a full snippet of the SQL 

### Dialect

Snowflake

### Version

0.13.0 and 0.11.1

### Configuration

`.sqlfluff`:
```
[sqlfluff]
templater = dbt
dialect = snowflake

[sqlfluff:templater:dbt]
project_dir = dbt/

# Defaults on anything not specified explicitly: https://docs.sqlfluff.com/en/stable/configuration.html#default-configuration
[sqlfluff:rules]
max_line_length = 120
comma_style = leading

# Keyword capitalisation
[sqlfluff:rules:L010]
capitalisation_policy = lower

# TODO: this supports pascal but not snake
# TODO: this inherits throwing violation on all unquoted identifiers... we can limit to aliases or column aliases
# [sqlfluff:rules:L014]
# extended_capitalisation_policy = pascal

# TODO: not 100% certain that this default is correct
# [sqlfluff:rules:L029]
## Keywords should not be used as identifiers.
# unquoted_identifiers_policy = aliases
# quoted_identifiers_policy = none
## Comma separated list of words to ignore for this rule
# ignore_words = None

# Function name capitalisation
[sqlfluff:rules:L030]
extended_capitalisation_policy = lower
```

### Are you willing to work on and submit a PR to address the issue?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)



Discussion:
I can't reproduce this, but this usually happens when the file itself is in some other format, rather than UTF-8, to begin with. Can you confirm it's definitely UTF-8 encoded? some tips here on how to check this: https://stackoverflow.com/questions/6947749/how-to-check-if-a-txt-file-is-in-ascii-or-utf-8-format-in-windows-environment
You'll probably need to explicitly set the encoding. SQLFluff defaults to using `autodetect`, which is implemented by the third-party `chardet` package, but it's not guaranteed to always do the right thing. If it misbehaves, we (SQLFluff) can't do anything about it.

```
# can either be autodetect or a valid encoding e.g. utf-8, utf-8-sig
encoding = autodetect
```

We'd like to hear back from you, but this issue is likely to be closed as "won't fix"/"can't fix"
I have confirmed that the file is indeed utf-8 encoded and I have explicitly set the encoding to utf-8 and retested with the same result.

After running `sqlfluff fix` I have seen the encoding change from utf-8 to western (Windows 1252)

EDIT: If i manually force the file to be utf-8 AFTER `sqlfluff fix`, it resolves the issue. Good to know, but not a sustainable solution
Did you set the encoding property in `.sqlfluff`? It does not appear in the `.sqlfluff` file you provided above.

Also, please provide a test SQL file. You only provided a comment, not a complete file. When I run `sqlfluff fix` on the file, I get:
```
(sqlfluff-3.9.1) ➜  sqlfluff git:(main) ✗ sqlfluff fix test.sql
==== finding fixable violations ====
==== no fixable linting violations found ====
All Finished 📜 🎉!
```
I did. The config file I provided does not contain it but I retested using your suggestion and had the same result
I tried the above on my Mac. The resulting file looked okay to me:
```
 - tariff scenario —> dm_tariff_scenario
```

What operating system are using? Windows? Mac? Linux?
I am on an intel mac with Montery 12.3.1

Are you able to run sqlfluff fix twice in succession? The first run is fine, its the second run that fails

(Depending on my editor, it may or may not show the offending character. ie vim shows it, sublime does not)

Yes, I can run it twice in succession. The first time, it fixes a bunch of things. The second time, no issues found. Partial output below.
```
L:  83 | P:  13 | L003 | Expected 0 indentations, found 3 [compared to line 01]
L:  84 | P:   5 | L003 | Expected 0 indentations, found 1 [compared to line 01]
==== fixing violations ====
72 fixable linting violations found
Are you sure you wish to attempt to fix these? [Y/n] ...
Attempting fixes...
Persisting Changes...
== [test.sql] PASS
Done. Please check your files to confirm.
All Finished 📜 🎉!
  [3 unfixable linting violations found]
(sqlfluff-3.9.1) ➜  sqlfluff git:(main) ✗ sqlfluff fix test.sql
==== finding fixable violations ====
==== no fixable linting violations found ====
All Finished 📜 🎉!
  [2 unfixable linting violations found]
```

I'm on an M1 Mac with Big Sur (11.5.2).

Very strange behavior:
* That I can't reproduce it on a similar machine
* That setting `encoding = utf-8` in `.sqlfluff` doesn't fix it.

Note that AFAIK, "encoding" is not a real property of most files file. It's a guess made when reading the file. Some file formats let you specify the encoding, but SQL is not one of them. Hence the need to use a package like `chardet`.

E.g. Python lets you do it with a special comment: https://stackoverflow.com/questions/6289474/working-with-utf-8-encoding-in-python-source
I just noticed interesting behavior. I ran with `-vv` to ensure my config and although I am specifying `encoding = utf-8`, the -vv output seems to suggest `autodetect`. It is honoring other config (like `dbt`). Attempting to see where I have gone wrong on my side

EDIT: for context on directory structure: 
```
.sqlfluff
./dbt/models/marts/core/file.sql
```
I am running sqlfluff from the same directory as the `.sqlfluff` file ie `sqlfluff fix dbt/models/marts/core/file.sql`
I've heard the behavior can become tricky if you have multiple .sqlfluff files in subdirectories, etc. Are you certain you added the setting in the correct section of the file? If you put it in the wrong place, it'll be ignored, and it'll use the default setting instead, which is autodetect.
it is at the top level 
```
[sqlfluff]
templater = dbt
dialect = snowflake
encoding = utf-8

...
```
as per your default configuration docs. There are no .sqlfluff files in sub folders in that directory
@barrywhart Okay... so if I specify `--encoding utf-8` as a CLI command I am able to fix the file with no issue!! Thank you for helping with that!

I am unsure why it is not honoring that config however. Is there a way you would recommend debugging this issue from my side? We use this both as a CLI tool and as a pre-commit - so we are able to use the `--encoding` option explicitly, but it provides peace of mind to know why it _seems_ to not honor specific configs

I have changed other configs (ie adding an `excluded_rule`) and it IS honoring that (with no other changes to how i am running it)

Also super appreciate all the help :) 
Let me look into it later (probably in the next day or two). Not many people use this option, so I'd like to double check that it's being read correctly from config.
awesome! I appreciate it @barrywhart (and @tunetheweb )!

We, as an organization, are investing in SQLFluff as our production linter and we appreciate your support!
Thanks for the kind words. It's exciting to us seeing the project catching on. I've been involved with the project since late 2019, and I'm proud of the progress it's made. It seems to be becoming pretty mainstream now. One reason I've stayed involved is, how often do you get to help invent a fundamental new industry tool? 😊

BTW, feel free to delete your example SQL from the issue. It seems like we may not need it anymore?
Exactly! I have been loosely following this project for the past year and have been pushing to use it widely for a while! We adopted DBT and, since SQLFluff interacts well with DBT, we got the buy-in to invest :)

And yes I will delete the SQL!

Please let me know what you find relating to the encoding configuration! I am continuing to fiddle from my side!
I'm seeing the same issue -- seems that the `encoding` setting in `.sqlfluff` is not being read correctly:
```
[sqlfluff]
encoding = utf-8
```

We have automated tests for encoding, but they are lower-level tests (i.e. they exercise internal code directly, not reading encoding from `.sqlfluff`).

I'll take a closer look. Presumably, this should be easy to fix.

Patch:
diff --git a/src/sqlfluff/cli/commands.py b/src/sqlfluff/cli/commands.py
--- a/src/sqlfluff/cli/commands.py
+++ b/src/sqlfluff/cli/commands.py
@@ -247,7 +247,7 @@ def core_options(f: Callable) -> Callable:
     )(f)
     f = click.option(
         "--encoding",
-        default="autodetect",
+        default=None,
         help=(
             "Specify encoding to use when reading and writing files. Defaults to "
             "autodetect."


Test patch:
diff --git a/test/cli/commands_test.py b/test/cli/commands_test.py
--- a/test/cli/commands_test.py
+++ b/test/cli/commands_test.py
@@ -1256,44 +1256,40 @@ def test_encoding(encoding_in, encoding_out):
         )
 
 
-def test_cli_pass_on_correct_encoding_argument():
+@pytest.mark.parametrize(
+    "encoding,method,expect_success",
+    [
+        ("utf-8", "command-line", False),
+        ("utf-8-SIG", "command-line", True),
+        ("utf-8", "config-file", False),
+        ("utf-8-SIG", "config-file", True),
+    ],
+)
+def test_cli_encoding(encoding, method, expect_success, tmpdir):
     """Try loading a utf-8-SIG encoded file using the correct encoding via the cli."""
+    sql_path = "test/fixtures/cli/encoding_test.sql"
+    if method == "command-line":
+        options = [sql_path, "--encoding", encoding]
+    else:
+        assert method == "config-file"
+        with open(str(tmpdir / ".sqlfluff"), "w") as f:
+            print(f"[sqlfluff]\ndialect=ansi\nencoding = {encoding}", file=f)
+        shutil.copy(sql_path, tmpdir)
+        options = [str(tmpdir / "encoding_test.sql")]
     result = invoke_assert_code(
         ret_code=65,
         args=[
             lint,
-            [
-                "test/fixtures/cli/encoding_test.sql",
-                "--encoding",
-                "utf-8-SIG",
-            ],
-        ],
-    )
-    raw_output = repr(result.output)
-
-    # Incorrect encoding raises paring and lexer errors.
-    assert r"L:   1 | P:   1 |  LXR |" not in raw_output
-    assert r"L:   1 | P:   1 |  PRS |" not in raw_output
-
-
-def test_cli_fail_on_wrong_encoding_argument():
-    """Try loading a utf-8-SIG encoded file using the wrong encoding via the cli."""
-    result = invoke_assert_code(
-        ret_code=65,
-        args=[
-            lint,
-            [
-                "test/fixtures/cli/encoding_test.sql",
-                "--encoding",
-                "utf-8",
-            ],
+            options,
         ],
     )
     raw_output = repr(result.output)
 
-    # Incorrect encoding raises paring and lexer errors.
-    assert r"L:   1 | P:   1 |  LXR |" in raw_output
-    assert r"L:   1 | P:   1 |  PRS |" in raw_output
+    # Incorrect encoding raises parsing and lexer errors.
+    success1 = r"L:   1 | P:   1 |  LXR |" not in raw_output
+    success2 = r"L:   1 | P:   1 |  PRS |" not in raw_output
+    assert success1 == expect_success
+    assert success2 == expect_success
 
 
 def test_cli_no_disable_noqa_flag():