Repository: sqlfluff/sqlfluff
Pull request: Normalise input newlines
<!--Firstly, thanks for adding this feature! Secondly, please check the key steps against the checklist below to make your contribution easy to merge.-->

<!--Please give the Pull Request a meaningful title (including the dialect this PR is for if it is dialect specific), as this will automatically be added to the release notes, and then the Change Log.-->

### Brief summary of the change made
<!--If there is an open issue for this, then please include `fixes #XXXX` or `closes #XXXX` replacing `XXXX` with the issue number and it will automatically close the issue when the pull request is merged. Alternatively if not fully closed you can say `makes progress on #XXXX` to create a link on that issue without closing it.-->
Fixes #797. After investigation (described in the issue) it seems most appropriate to not handle multiple newlines and instead add step to ensure all newline inputs to sqlfluff are normalised to unix-style line endings for consistency. Users wanting newline conversion should consider the easy to use [mixed-line-ending](https://github.com/pre-commit/pre-commit-hooks/blob/master/pre_commit_hooks/mixed_line_ending.py) pre-commit hook or the `dos2unix` command. 

Currently we load from files using universal newlines, but we don't do this for test input. Therefore I have added a conversion function to the linter to ensure consistency regardless of how the SQL is supplied.

### Are there any other side effects of this change that we should be aware of?
No

### Pull Request checklist
- [X] Please confirm you have completed any of the necessary steps below.

- Included test cases to demonstrate any code changes, which may be one or more of the following:
  - `.yml` rule test cases in `test/fixtures/rules/std_rule_cases`.
  - `.sql`/`.yml` parser test cases in `test/fixtures/dialects` (note YML files can be auto generated with `python test/generate_parse_fixture_yml.py` or by running `tox` locally).
  - Full autofix test cases in `test/fixtures/linter/autofix`.
  - Other.
- Added appropriate documentation for the change.
- Created GitHub issues for any relevant followup/future enhancements if appropriate.


Problem statement:
Feature request: Enable use of unix-style line endings when running sqlfluff fix command on Windows
In our environment, we use Windows PCs for development, but the company standard is that SQL files have unix-style line endings (LF).  We use bash under WSL a lot, and set Windows editors to use LF only.  However, the only python we have available is on Windows. When running sqlfluff fix under python on Windows, the file gets changed from LF to Windows-style line endings CRLF.

I understand that this is python default behavior, but it would be nice to be able to over-ride the default and choose LF only, all the time.

Hints from issue discussion:
I think the answer to this is just a config option to specify line endings when writing files, with options of maybe `system default`, LF and CRLF?
A config option would be perfect.  For completeness, may as well include CR for Classic Mac, although probably not use much any more.
I'm going to take a look at adding this as I've got mac and windows users on my team and hate seeing those carriage returns :) 

@NiallRees's config option suggestion is definitely the right approach.


FYI the config option only approach was not appropriate on further investigation (see closed pr) but have a rule based approach will be more robust

Commit message: Merge branch 'main' into normalise_newlines

Patch to review:
diff --git a/src/sqlfluff/core/linter/linter.py b/src/sqlfluff/core/linter/linter.py
index 0c96b7f338b..8c110439757 100644
--- a/src/sqlfluff/core/linter/linter.py
+++ b/src/sqlfluff/core/linter/linter.py
@@ -16,7 +16,7 @@
 )
 
 import pathspec
-
+import regex
 from tqdm import tqdm
 
 from sqlfluff.core.errors import (
@@ -116,6 +116,11 @@ def _load_raw_file_and_config(
         # Return the raw file and config
         return raw_file, file_config, encoding
 
+    @staticmethod
+    def _normalise_newlines(string: str) -> str:
+        """Normalise newlines to unix-style line endings."""
+        return regex.sub(r"\n|\r\n|\r", "\n", string)
+
     @staticmethod
     def _lex_templated_file(
         templated_file: TemplatedFile, config: FluffConfig
@@ -606,6 +611,11 @@ def render_string(
         # Start the templating timer
         t0 = time.monotonic()
 
+        # Newlines are normalised to unix-style line endings (\n).
+        # The motivation is that Jinja normalises newlines during templating and
+        # we want consistent mapping between the raw and templated slices.
+        in_str = self._normalise_newlines(in_str)
+
         if not config.get("templater_obj") == self.templater:
             linter_logger.warning(
                 (
diff --git a/test/core/linter_test.py b/test/core/linter_test.py
index f1e57b9fabe..40ca7b81453 100644
--- a/test/core/linter_test.py
+++ b/test/core/linter_test.py
@@ -885,3 +885,10 @@ def test_advanced_api_methods():
 
     # BaseFileSegment.get_table_references & StatementSegment.get_table_references
     assert parsed.tree.get_table_references() == {"tab_a", "tab_b"}
+
+
+def test_normalise_newlines():
+    """Test normalising newlines to unix-style line endings."""
+    in_str = "SELECT\r\n foo\n FROM \r \n\r bar;"
+    out_str = "SELECT\n foo\n FROM \n \n\n bar;"
+    assert out_str == Linter._normalise_newlines(in_str)


Code review comments:
diff_hunk: @@ -116,6 +116,11 @@ def _load_raw_file_and_config(
         # Return the raw file and config
         return raw_file, file_config, encoding
 
+    @staticmethod
+    def _normalise_newlines(string: str) -> str:
+        """Normalise newlines to unix-style line endings."""
+        return regex.sub(r"\n|\r\n|\r", "\n", string)
path: src/sqlfluff/core/linter/linter.py
text: @user1:
Do we need this first one? Seems to be replacing `"\n"` with `"\n"`?

```suggestion
        return regex.sub(r"\r\n|\r", "\n", string)
```

@author:
strictly no, but thought it was clearer to read as replacing all newlines with the unix newline (i.e. normalise), although in this case it's not like we'll ever change the newline replace style so happy either way.

I'll let you commit the suggestion and merge if you want