Repository: sqlfluff/sqlfluff

Issue:
Feature request: Enable use of unix-style line endings when running sqlfluff fix command on Windows
In our environment, we use Windows PCs for development, but the company standard is that SQL files have unix-style line endings (LF).  We use bash under WSL a lot, and set Windows editors to use LF only.  However, the only python we have available is on Windows. When running sqlfluff fix under python on Windows, the file gets changed from LF to Windows-style line endings CRLF.

I understand that this is python default behavior, but it would be nice to be able to over-ride the default and choose LF only, all the time.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) Report
> Merging [#2128](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) (afe6d0d) into [main](https://codecov.io/gh/sqlfluff/sqlfluff/commit/bf12e8b2371fdcfc617282256c2d9d2d4d9e9f7e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) (bf12e8b) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128/graphs/tree.svg?width=650&height=150&src=pr&token=maM8kLgCmn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)

```diff
@@             Coverage Diff             @@
##              main    #2128      +/-   ##
===========================================
- Coverage   100.00%   99.99%   -0.01%     
===========================================
  Files          148      148              
  Lines        10646    10651       +5     
===========================================
+ Hits         10646    10650       +4     
- Misses           0        1       +1     
```


| [Impacted Files](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) | Coverage Δ | |
|---|---|---|
| [src/sqlfluff/core/linter/linter.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff#diff-c3JjL3NxbGZsdWZmL2NvcmUvbGludGVyL2xpbnRlci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/sqlfluff/core/config.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff#diff-c3JjL3NxbGZsdWZmL2NvcmUvY29uZmlnLnB5) | `99.67% <0.00%> (-0.33%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff). Last update [bf12e8b...afe6d0d](https://codecov.io/gh/sqlfluff/sqlfluff/pull/2128?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff).

tunetheweb: Isn’t there a concern here that `sqlfluff fix` will now effectively rewrite line endings and lead to massive diffs every time a file needs a fix? At least until the code base is converted to unix style newlines in entirety?

Note I’m not talking about our code base (that I already converted) but any other users.
jpy-git: @tunetheweb python already automatically does this when it reads in files so it's no different from what we already do, I'm just making it consistent for the API and CLI stdin.
jpy-git: also when python writes out a file it automatically converts them back to the appropriate line ending for the system
jpy-git: @barrywhart follow up PR to my post in #797 
tunetheweb: > @tunetheweb python already automatically does this when it reads in files so it's no different from what we already do, I'm just making it consistent for the API and CLI stdin.

I'm confused. If python already automatically does this, then why's there even an issue? Sorry if I'm missing the obvious here...
jpy-git: @tunetheweb because it only does it when reading in from a file (i.e. `open("<some_sql_path>", "r") as f`), but not when we read the sql from either stdin in the CLI or via the string input to the API. So we should normalise those too for consistency
tunetheweb: OK. Thanks for explaining.

Patch:
diff --git a/src/sqlfluff/core/linter/linter.py b/src/sqlfluff/core/linter/linter.py
index 0c96b7f33..a720700c6 100644
--- a/src/sqlfluff/core/linter/linter.py
+++ b/src/sqlfluff/core/linter/linter.py
@@ -16,7 +16,7 @@ from typing import (
 )
 
 import pathspec
-
+import regex
 from tqdm import tqdm
 
 from sqlfluff.core.errors import (
@@ -116,6 +116,11 @@ class Linter:
         # Return the raw file and config
         return raw_file, file_config, encoding
 
+    @staticmethod
+    def _normalise_newlines(string: str) -> str:
+        """Normalise newlines to unix-style line endings."""
+        return regex.sub(r"\r\n|\r", "\n", string)
+
     @staticmethod
     def _lex_templated_file(
         templated_file: TemplatedFile, config: FluffConfig
@@ -606,6 +611,11 @@ class Linter:
         # Start the templating timer
         t0 = time.monotonic()
 
+        # Newlines are normalised to unix-style line endings (\n).
+        # The motivation is that Jinja normalises newlines during templating and
+        # we want consistent mapping between the raw and templated slices.
+        in_str = self._normalise_newlines(in_str)
+
         if not config.get("templater_obj") == self.templater:
             linter_logger.warning(
                 (


Test patch:
diff --git a/test/core/linter_test.py b/test/core/linter_test.py
index f1e57b9fa..40ca7b814 100644
--- a/test/core/linter_test.py
+++ b/test/core/linter_test.py
@@ -885,3 +885,10 @@ def test_advanced_api_methods():
 
     # BaseFileSegment.get_table_references & StatementSegment.get_table_references
     assert parsed.tree.get_table_references() == {"tab_a", "tab_b"}
+
+
+def test_normalise_newlines():
+    """Test normalising newlines to unix-style line endings."""
+    in_str = "SELECT\r\n foo\n FROM \r \n\r bar;"
+    out_str = "SELECT\n foo\n FROM \n \n\n bar;"
+    assert out_str == Linter._normalise_newlines(in_str)