Repository: sqlfluff/sqlfluff

Issue:
"Applying rule L016 threw an Exception: Is this possible?"
## Expected Behaviour

Probably valid SQL should not cause internal exceptions in SQLFluff.

## Observed Behaviour

`sqlfluff fix` prints an exception. However, that does not prevent in from working as a whole, so this does not block general usage.

## Steps to Reproduce

```console
$ echo "{% set elements = 'foo' %}

    SELECT CASE
      {% for elem in elements %}
        WHEN '{{ elem }}' = '' THEN 1
        WHEN '{{ elem }}' = '' THEN 1
      {%endfor %}
      END

" | sqlfluff fix -
CRITICAL   [L016] Applying rule L016 threw an Exception: Is this possible? 
Traceback (most recent call last):
  File "/home/vladimir/work/extern/sqlfluff/src/sqlfluff/core/rules/base.py", line 302, in crawl
    res = self._eval(
  File "/home/vladimir/work/extern/sqlfluff/src/sqlfluff/rules/L016.py", line 511, in _eval
    fixes = self._eval_line_for_breaks(this_line)
  File "/home/vladimir/work/extern/sqlfluff/src/sqlfluff/rules/L016.py", line 266, in _eval_line_for_breaks
    raise ValueError("Is this possible?")  # pragma: no cover
ValueError: Is this possible?
(...repeated 6 times...)
Unfixable violations detected.
```

## Dialect

None specified

## Version
SQLFluff 6011bdbe05669b075045e8127cdf18cc537686d4, Python 3.9.6

## Configuration

None

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) Report
> Merging [#1494](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) (3a406a2) into [main](https://codecov.io/gh/sqlfluff/sqlfluff/commit/826798198aa663739ad81ce7b1b3e4192b4d4ab9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) (8267981) will **not change** coverage.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494/graphs/tree.svg?width=650&height=150&src=pr&token=maM8kLgCmn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)

```diff
@@            Coverage Diff            @@
##              main     #1494   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          130       130           
  Lines         8849      8860   +11     
=========================================
+ Hits          8849      8860   +11     
```


| [Impacted Files](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) | Coverage Δ | |
|---|---|---|
| [src/sqlfluff/core/parser/lexer.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGFyc2VyL2xleGVyLnB5) | `100.00% <ø> (ø)` | |
| [src/sqlfluff/core/templaters/base.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff#diff-c3JjL3NxbGZsdWZmL2NvcmUvdGVtcGxhdGVycy9iYXNlLnB5) | `100.00% <ø> (ø)` | |
| [src/sqlfluff/core/templaters/python.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff#diff-c3JjL3NxbGZsdWZmL2NvcmUvdGVtcGxhdGVycy9weXRob24ucHk=) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff). Last update [8267981...3a406a2](https://codecov.io/gh/sqlfluff/sqlfluff/pull/1494?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff).

tunetheweb: OK this is just impossible to fix. So I went with the plain and simple option of ignoring newline segments in rule L016.

Can you have a look @barrywhart ?
barrywhart: Looks good!

Patch:
diff --git a/src/sqlfluff/rules/L016.py b/src/sqlfluff/rules/L016.py
index 2e30e3ccc..0643e3d69 100644
--- a/src/sqlfluff/rules/L016.py
+++ b/src/sqlfluff/rules/L016.py
@@ -376,6 +376,15 @@ class Rule_L016(Rule_L003):
             # them. Rationale: The intent of this rule is to enforce maximum
             # line length, and newlines don't make lines longer.
             return 0
+
+        if "\n" in segment.pos_marker.source_str():
+            # Similarly we shouldn't see newlines in source segments
+            # However for templated loops it's often not possible to
+            # accurately calculate the segments. These will be caught by
+            # the first iteration of the loop (which is non-templated)
+            # so doesn't suffer from the same bug, so we can ignore these
+            return 0
+
         # Compute the length of this segments in SOURCE space (before template
         # expansion).
         slice_length = (


Test patch:
diff --git a/test/fixtures/rules/std_rule_cases/L016.yml b/test/fixtures/rules/std_rule_cases/L016.yml
index 7ce400c33..ec746bf4e 100644
--- a/test/fixtures/rules/std_rule_cases/L016.yml
+++ b/test/fixtures/rules/std_rule_cases/L016.yml
@@ -156,3 +156,77 @@ test_long_jina_comment_ignore:
       max_line_length: 80
       L016:
         ignore_comment_lines: True
+
+
+test_for_loop:
+  # A Jinja for loop
+  pass_str: |
+    {% for elem in 'foo' %}
+    SELECT '{{ elem }}' FROM table1;
+    SELECT '{{ elem }}' FROM table2;
+    {% endfor %}
+
+
+test_for_loop_repeating_elements_starts_with_literal:
+  # A Jinja for loop with repeating elements (that are difficult to match)
+  # but starting with a literal that can be used to match
+  pass_str: |
+    {% set elements = 'foo' %}
+    SELECT
+        CASE
+            {% for elem in elements %}
+            WHEN '{{ elem }}' = '' THEN 1
+            WHEN '{{ elem }}' = '' THEN 1
+            {% endfor %}
+        END
+
+
+test_for_loop_starting_with_templated_piece:
+  # A Jinja for loop starting with non-literals
+  # But unique parts can be used to match
+  pass_str: |
+    {% set elements = 'foo' %}
+    {% set when = 'WHEN' %}
+    SELECT
+        CASE
+            {% for elem in elements %}
+            {{ when }} '{{ elem }}' = '' THEN 1
+            {{ when }} '{{ elem }}' = '' THEN 2
+            {% endfor %}
+        END
+
+test_for_loop_fail_complex_match:
+  # A Jinja for loop starting with non-literals
+  # But non-unique parts which therefore cannot
+  # be used to match
+  pass_str: |
+    {% set elements = 'foo' %}
+    {% set when = 'WHEN' %}
+    SELECT
+        CASE
+            {% for elem in elements %}
+            {{ when }} '{{ elem }}' = '' THEN 1
+            {{ when }} '{{ elem }}' = '' THEN 1
+            {% endfor %}
+        END
+
+test_for_loop_fail_simple_match:
+  # If for loop only contains literals it should still pass
+  pass_str: |
+    {% set elements = 'foo' %}
+    SELECT
+        CASE
+            {% for elem in elements %}
+            WHEN 'f' THEN a
+            {% endfor %}
+        END
+
+
+test_set_statement:
+  # A Jinja set statement
+  pass_str: |
+    {% set statement = "SELECT 1 from table1;" %}
+    {{ statement }}{{ statement }}
+  configs:
+    rules:
+      max_line_length: 80
diff --git a/test/fixtures/templater/jinja_l_metas/002.yml b/test/fixtures/templater/jinja_l_metas/002.yml
index 2c501dbbf..397f57ecd 100644
--- a/test/fixtures/templater/jinja_l_metas/002.yml
+++ b/test/fixtures/templater/jinja_l_metas/002.yml
@@ -184,4 +184,4 @@ file:
                 table_reference:
                   identifier: action_states
           - dedent: ''
-  newline: "\n"
\ No newline at end of file
+  newline: "\n"