Repository: sqlfluff/sqlfluff
Pull request: Robust Jinja raw/template mapping
### Brief summary of the change made

Fixes #1437

There are two main aspects to the design:
* During slicing, creates a second Jinja template that is **structurally** identical, but literals and templated blocks are replaced and/or tagged with UUIDs. Using this alternate template, all template output can be fully and reliably mapped back to the corresponding raw `literal` or `templated` slices.
* A new class, `TemplateTracer`, uses a "flowchart" of the template's execution structure to deduce the other types of `TemplatedFileSlice`s besides `literal` and `templated` (`if`, `for`, etc.).

It achieves this without requiring changes to the third-party Jinja package.

### Detailed inventory of changes

- `core/parser`
  - `sequence.py`: When deciding whether to add indents, ignore those where `template`=`True`
  - `lexer.py`: With multiple source-only slices, convert to, e.g. `{% else %}... 50 omitted ... {% endif %}`. May be affecting `L003` template indentation.
    - Split into 3 placeholders?
    - Remove entirely?
    - Other?
  -   `meta.py`: Segments like `Indent` can now be marked `template` = `True`, used now by `sequence.py`
-   `core/rules`
  - `base.py`: Compatibility tweak to continue working in spite of added `RawFIleSlice` fields
- `core/templaters`
  - `base.py`
    - `RawFileSlice`
       - Add 3 fields: `unique_alternate_id`, `alternate_code`, `next_slice_indices`
      - Add `as_tuple()` method to access *just* the old fields
      - Changed from `NamedTuple` to `dataclass`
       - Various compatibility tweaks related to `RawFileSlice` changes
  - `jinja.py`
    - Override `slice_file()` from base class `PythonTemplater`, uses `TemplateTracer`
    - New class `TemplateTracer` that emulates template execution based on output from alternate template
  - `python.py`
    - Various compatibility tweaks related to `RawFileSlice` changes
- `rules`
  - `L003.py`
    - Deactivate `_reorder_raw_stack()`: Unneeded now?
    - `_coerce_indent_to()`: Enhance to support *removing* whitespace
    - `_process_current_line()`: New custom logic to handle template blocks on a line by themselves. Handles some cases nicely but handling tricky cases may become a "swamp". 😬
    -   Added helper functions to support `_process_current_line()` changes

### Are there any other side effects of this change that we should be aware of?

This is a fundamental change to the Jinja raw/templated slice mapping. For now, it is designed to leave the other templaters (Python and dbt) "as is". At some point in the future, I expect we'll want to look at updating dbt to use this design. 

### Pull Request checklist
- [X] Please confirm you have completed any of the necessary steps below.

- Included test cases to demonstrate any code changes, which may be one or more of the following:
  - `.yml` rule test cases in `test/fixtures/rules/std_rule_cases`.
  - `.sql`/`.yml` parser test cases in `test/fixtures/dialects` (note YML files can be auto generated with `python test/generate_parse_fixture_yml.py` or by running `tox` locally).
  - Full autofix test cases in `test/fixtures/linter/autofix`.
  - Other.
- Added appropriate documentation for the change.
- Created GitHub issues for any relevant followup/future enhancements if appropriate.


Problem statement:
Jinja: Can we do a more robust mapping between expanded code and original templated code?
SQLFluff has a hard time applying fixes without getting deeper into Jinja. As Alan mentioned in a recent interview, Jinja doesn't seem to support this. I have a suspicion that if we look really closely at Jinja, we may be able to do some things to make this work, although they may be hacky at first. But if we find something promising, maybe we can submit an upstream PR.

Basically, if we could determine which sections of code came from different iterations of a loop, we could avoid these issues much more reliably and straightforwardly.

Hints from issue discussion:
This looks like a good template to experiment with. It contains 3 copies of the same loop. `sqlfluff fix` refuses to fix it -- it appears to be discarding all the fixes because it can't map them to the source. (I say this because if I remove two copies of the loop, it is able to fix the file.)
```
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: edit @[L:  4, P:  7] edt:'  '->'    '>]
INFO             * Discarding fixes that span blocks: [<LintFix: delete @[L:  4, P:  7] delete:'brand'>, <LintFix: delete @[L:  4, P:  7] delete:'  '>, <LintFix: delete @[L:  4, P:  7] delete:'\n'>, <LintFix: edit @[L:  4, P:  7] edt:'\n'->' brand\n'>]
INFO             * Discarding fixes that span blocks: [<LintFix: delete @[L:  4, P:  7] delete:'brand'>, <LintFix: delete @[L:  4, P:  7] delete:'  '>, <LintFix: delete @[L:  4, P:  7] delete:'\n'>, <LintFix: edit @[L:  4, P:  7] edt:'\n'->' brand\n'>]
INFO             * Discarding fixes that span blocks: [<LintFix: delete @[L:  4, P:  7] delete:'brand'>, <LintFix: delete @[L:  4, P:  7] delete:'  '>, <LintFix: delete @[L:  4, P:  7] delete:'\n'>, <LintFix: edit @[L:  4, P:  7] edt:'\n'->' brand\n'>]
INFO             * Discarding fixes that span blocks: [<LintFix: delete @[L:  4, P:  7] delete:'brand'>, <LintFix: delete @[L:  4, P:  7] delete:'  '>, <LintFix: delete @[L:  4, P:  7] delete:'\n'>, <LintFix: edit @[L:  4, P:  7] edt:'\n'->' brand\n'>]
INFO             * Discarding fixes that span blocks: [<LintFix: delete @[L:  4, P:  7] delete:'brand'>, <LintFix: delete @[L:  4, P:  7] delete:'  '>, <LintFix: delete @[L:  4, P:  7] delete:'\n'>, <LintFix: edit @[L:  4, P:  7] edt:'\n'->' brand\n'>]
INFO             * Discarding fixes that span blocks: [<LintFix: delete @[L:  4, P:  7] delete:'brand'>, <LintFix: delete @[L:  4, P:  7] delete:'  '>, <LintFix: delete @[L:  4, P:  7] delete:'\n'>, <LintFix: edit @[L:  4, P:  7] edt:'\n'->' brand\n'>]
INFO       Fix loop complete. Stability achieved after 0/10 loops.
== [test.sql] PASS
==== no fixable linting violations found ====

```

Template:
```
{% set products =  [
  'table1',
  'table2',
  ] %}

{% for product in products %}
SELECT
  brand
FROM
  {{ product }}
{% if not loop.last -%} UNION ALL {%- endif %}
{% endfor %}
;

{% for product in products %}
SELECT
  brand
FROM
  {{ product }}
{% if not loop.last -%} UNION ALL {%- endif %}
{% endfor %}
;

{% for product in products %}
SELECT
  brand
FROM
  {{ product }}
{% if not loop.last -%} UNION ALL {%- endif %}
{% endfor %}
```

Commit message: More work on L003

Patch to review:
diff --git a/src/sqlfluff/core/parser/grammar/sequence.py b/src/sqlfluff/core/parser/grammar/sequence.py
index 580156fee8d..dc27134f0dd 100644
--- a/src/sqlfluff/core/parser/grammar/sequence.py
+++ b/src/sqlfluff/core/parser/grammar/sequence.py
@@ -337,7 +337,11 @@ def match(
             # Have we already got indents?
             meta_idx = None
             for idx, seg in enumerate(bracket_segment.segments):
-                if seg.is_meta and cast(MetaSegment, seg).indent_val > 0:
+                if (
+                    seg.is_meta
+                    and cast(MetaSegment, seg).indent_val > 0
+                    and not seg.template
+                ):
                     meta_idx = idx
                     break
             # If we've already got indents, don't add more.
diff --git a/src/sqlfluff/core/parser/lexer.py b/src/sqlfluff/core/parser/lexer.py
index de4ff54fa53..ff060687941 100644
--- a/src/sqlfluff/core/parser/lexer.py
+++ b/src/sqlfluff/core/parser/lexer.py
@@ -401,6 +401,51 @@ def elements_to_segments(
                 # only one of them.
                 if not placeholder_str:
                     placeholder_str = "".join(s.raw for s in so_slices)
+                # The Jinja templater sometimes returns source-only slices with
+                # gaps between. For example, in this section:
+                #
+                #   {% else %}
+                #   JOIN
+                #       {{action}}_raw_effect_sizes
+                #   USING
+                #       ({{ states }})
+                #   {% endif %}
+                #
+                # , we might get {% else %} and {% endif %} slices, without the
+                # 4 lines between. This indicates those lines were not executed
+                # In this case, generate a placeholder where the skipped code is
+                # omitted but noted with a brief string, e.g.:
+                #
+                # "{% else %}... [103 omitted] ...{% endif %}".
+                #
+                # This is more readable -- it would be REALLY confusing for a
+                # placeholder to include code that wasn't even executed!!
+                if len(so_slices) >= 2:
+                    has_gap = False
+                    gap_placeholder_parts = []
+                    last_slice = None
+                    # For each slice...
+                    for so_slice in so_slices:
+                        # If it's not the first slice, was there a gap?
+                        if last_slice:
+                            end_last = last_slice.source_idx + len(last_slice.raw)
+                            num_omitted = so_slice.source_idx - end_last
+                            if num_omitted:
+                                # Yes, gap between last_slice and so_slice.
+                                has_gap = True
+
+                                # Generate a string documenting the gap.
+                                if num_omitted >= 10:
+                                    gap_placeholder_parts.append(
+                                        f"... [{num_omitted} omitted] ..."
+                                    )
+                                else:
+                                    gap_placeholder_parts.append("...")
+                        # Now add the slice's source.
+                        gap_placeholder_parts.append(so_slice.raw)
+                        last_slice = so_slice
+                    if has_gap:
+                        placeholder_str = "".join(gap_placeholder_parts)
                 lexer_logger.debug(
                     "    Overlap Length: %s. PS: %s, LS: %s, p_str: %r, templ_str: %r",
                     existing_len,
@@ -410,7 +455,7 @@ def elements_to_segments(
                     templ_str,
                 )
 
-                # Caluculate potential indent/dedent
+                # Calculate potential indent/dedent
                 block_slices = sum(s.slice_type.startswith("block_") for s in so_slices)
                 block_balance = sum(
                     s.slice_type == "block_start" for s in so_slices
@@ -458,19 +503,20 @@ def elements_to_segments(
                     )
                 )
                 lexer_logger.debug(
-                    "      Placholder: %s, %r", segment_buffer[-1], placeholder_str
+                    "      Placeholder: %s, %r", segment_buffer[-1], placeholder_str
                 )
 
-                # Add a dedent if appropriate.
+                # Add an indent if appropriate.
                 if trail_indent and add_indents:
                     lexer_logger.debug("      INDENT")
                     segment_buffer.append(
                         Indent(
+                            template=True,
                             pos_marker=PositionMarker.from_point(
                                 placeholder_slice.stop,
                                 element.template_slice.start,
                                 templated_file,
-                            )
+                            ),
                         )
                     )
 
diff --git a/src/sqlfluff/core/parser/segments/meta.py b/src/sqlfluff/core/parser/segments/meta.py
index 8674fe6e4ff..fadc9333ee0 100644
--- a/src/sqlfluff/core/parser/segments/meta.py
+++ b/src/sqlfluff/core/parser/segments/meta.py
@@ -15,6 +15,10 @@ class MetaSegment(RawSegment):
     indent_val = 0
     is_meta = True
 
+    def __init__(self, template=False, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.template = template
+
     @staticmethod
     def _suffix():
         """Return any extra output required at the end when logging.
diff --git a/src/sqlfluff/core/rules/base.py b/src/sqlfluff/core/rules/base.py
index 91a9c88db1f..e5cd6429b09 100644
--- a/src/sqlfluff/core/rules/base.py
+++ b/src/sqlfluff/core/rules/base.py
@@ -496,7 +496,8 @@ def discard_unsafe_fixes(
         for fix in lint_result.fixes:
             if fix.anchor:
                 fix_slices.update(
-                    templated_file.raw_slices_spanning_source_slice(
+                    raw_slice.as_tuple()
+                    for raw_slice in templated_file.raw_slices_spanning_source_slice(
                         fix.anchor.pos_marker.source_slice
                     )
                 )
diff --git a/src/sqlfluff/core/templaters/base.py b/src/sqlfluff/core/templaters/base.py
index 1b28b0130b7..5e8ee7e142c 100644
--- a/src/sqlfluff/core/templaters/base.py
+++ b/src/sqlfluff/core/templaters/base.py
@@ -3,6 +3,7 @@
 import logging
 from bisect import bisect_left
 from collections import defaultdict
+from dataclasses import dataclass, field
 from typing import Dict, Iterator, List, Tuple, Optional, NamedTuple, Iterable
 
 from cached_property import cached_property
@@ -23,13 +24,17 @@ def iter_indices_of_newlines(raw_str: str) -> Iterator[int]:
             break  # pragma: no cover TODO?
 
 
-class RawFileSlice(NamedTuple):
+@dataclass
+class RawFileSlice:
     """A slice referring to a raw file."""
 
     raw: str
     slice_type: str
     source_idx: int
     slice_subtype: Optional[str] = None
+    unique_alternate_id: Optional[str] = None
+    alternate_code: Optional[str] = None
+    next_slice_indices: Optional[List[int]] = field(default_factory=list)
 
     def end_source_idx(self):
         """Return the closing index of this slice."""
@@ -39,6 +44,13 @@ def source_slice(self):
         """Return the a slice object for this slice."""
         return slice(self.source_idx, self.end_source_idx())
 
+    def as_tuple(self):
+        """Return a tuple version of this object.
+
+        This tuple contains a "public" subset of the fields.
+        """
+        return (self.raw, self.slice_type, self.source_idx, self.slice_subtype)
+
 
 class TemplatedFileSlice(NamedTuple):
     """A slice referring to a templated file."""
@@ -194,7 +206,7 @@ def raw_slice_block_info(self) -> RawSliceBlockInfo:
             if raw_slice.slice_type == "block_start":
                 blocks.append(raw_slice)
                 templater_logger.info("%d -> %r", block_id, raw_slice.raw)
-                block_ids[raw_slice] = block_id
+                block_ids[raw_slice.as_tuple()] = block_id
                 block_id += 1
                 if raw_slice.slice_subtype == "loop":
                     loops.add(block_id)
@@ -202,10 +214,10 @@ def raw_slice_block_info(self) -> RawSliceBlockInfo:
                 blocks.pop()
                 block_id += 1
                 templater_logger.info("%d -> %r", block_id, raw_slice.raw)
-                block_ids[raw_slice] = block_id
+                block_ids[raw_slice.as_tuple()] = block_id
             else:
                 templater_logger.info("%d -> %r", block_id, raw_slice.raw)
-                block_ids[raw_slice] = block_id
+                block_ids[raw_slice.as_tuple()] = block_id
         literal_only_loops = [
             block_id
             for block_id in set(block_ids.values())
@@ -359,17 +371,17 @@ def is_source_slice_literal(self, source_slice: slice) -> bool:
         if source_slice.start == source_slice.stop:
             return True
         is_literal = True
-        for _, seg_type, seg_idx, _ in self.raw_sliced:
+        for raw_slice in self.raw_sliced:
             # Reset if we find a literal and we're up to the start
             # otherwise set false.
-            if seg_idx <= source_slice.start:
-                is_literal = seg_type == "literal"
-            elif seg_idx >= source_slice.stop:
+            if raw_slice.source_idx <= source_slice.start:
+                is_literal = raw_slice.slice_type == "literal"
+            elif raw_slice.source_idx >= source_slice.stop:
                 # We've gone past the end. Break and Return.
                 break
             else:
                 # We're in the middle. Check type
-                if seg_type != "literal":
+                if raw_slice.slice_type != "literal":
                     is_literal = False
         return is_literal
 
diff --git a/src/sqlfluff/core/templaters/jinja.py b/src/sqlfluff/core/templaters/jinja.py
index 38d2c41eb54..c5a9834dc6d 100644
--- a/src/sqlfluff/core/templaters/jinja.py
+++ b/src/sqlfluff/core/templaters/jinja.py
@@ -4,7 +4,9 @@
 import logging
 import importlib.util
 import re
-from typing import Iterator, Tuple, Optional
+import uuid
+from itertools import chain
+from typing import List, Tuple, Optional
 
 from jinja2.sandbox import SandboxedEnvironment
 from jinja2 import meta, TemplateSyntaxError, TemplateError
@@ -15,6 +17,7 @@
 from sqlfluff.core.templaters.base import (
     TemplatedFile,
     RawFileSlice,
+    TemplatedFileSlice,
 )
 from sqlfluff.core.templaters.python import PythonTemplater
 
@@ -162,7 +165,7 @@ def _crawl_tree(cls, tree, variable_names, raw):
             )
 
     @staticmethod
-    def _get_jinja_env():
+    def get_jinja_env():
         """Get a properly configured jinja environment."""
         # We explicitly want to preserve newlines.
         return SandboxedEnvironment(
@@ -215,7 +218,7 @@ def process(
                 if name not in live_context:
                     live_context[name] = dbt_builtins[name]
 
-        env = self._get_jinja_env()
+        env = self.get_jinja_env()
 
         # Load macros from path (if applicable)
         macros_path = config.get_section(
@@ -233,9 +236,12 @@ def process(
 
         live_context.update(self._extract_libraries_from_config(config=config))
 
+        def make_template(in_str):
+            return env.from_string(in_str, globals=live_context)
+
         # Load the template, passing the global context.
         try:
-            template = env.from_string(in_str, globals=live_context)
+            template = make_template(in_str)
         except TemplateSyntaxError as err:
             # Something in the template didn't parse, return the original
             # and a violation around what happened.
@@ -277,7 +283,10 @@ def process(
             out_str = template.render()
             # Slice the file once rendered.
             raw_sliced, sliced_file, out_str = self.slice_file(
-                in_str, out_str, config=config
+                in_str,
+                out_str,
+                config=config,
+                make_template=make_template,
             )
             return (
                 TemplatedFile(
@@ -301,16 +310,176 @@ def process(
             )
             return None, violations
 
-    re_open_tag = re.compile(r"^\s*{%[\+\-]?\s*")
-    re_close_tag = re.compile(r"\s*[\+\-]?%}\s*$")
+    @classmethod
+    def slice_file(
+        cls, raw_str: str, templated_str: str, config=None, **kwargs
+    ) -> Tuple[List[RawFileSlice], List[TemplatedFileSlice], str]:
+        """Slice the file to determine regions where we can fix."""
+        make_template = kwargs.pop("make_template", None)
+        if make_template is None:
+            return super().slice_file(raw_str, templated_str, config, **kwargs)
+
+        templater_logger.info("Slicing File Template")
+        templater_logger.debug("    Raw String: %r", raw_str)
+        templater_logger.debug("    Templated String: %r", templated_str)
+        tracer = TemplateTracer(raw_str, templated_str, make_template)
+        tracer.process()
+        # for ts in tracer.sliced_file:
+        #     print(
+        #         f"{ts.slice_type} templated:{templated_str[ts.templated_slice]!r} raw:{raw_str[ts.source_slice]!r}"
+        #     )
+        return tracer.raw_sliced, tracer.sliced_file, templated_str
+
+
+class TemplateTracer:
+    """Deduces and records execution path of a Jinja template."""
+
+    re_open_tag = re.compile(r"^\s*({[{%])[\+\-]?\s*")
+    re_close_tag = re.compile(r"\s*[\+\-]?([}%]})\s*$")
+
+    def __init__(self, raw_str, templated_str, make_template):
+        self.raw_str = raw_str
+        self.templated_str = templated_str
+        self.make_template = make_template
+        self.program_counter = 0
+        self.raw_sliced = self._slice_template(self.raw_str)
+        self.sliced_file = []
+        self.source_idx = 0
+
+    def process(self):
+        """Captures the trace."""
+        alternate_template = "".join(
+            rs.alternate_code if rs.alternate_code is not None else rs.raw
+            for rs in self.raw_sliced
+        )
+        # print(alternate_template)
+        unique_alternate = self.make_template(alternate_template)
+        template = self.make_template(self.raw_str)
+        output1 = template.render()
+        output2 = unique_alternate.render()
+        parts = []
+        # Ideally, we'd like to receive one section of Jinja output at a
+        # time. In practice, Jinja internals determine how many section we
+        # get at a time. Here, we split s2 into one string per section.
+        # Each section is one of:
+        # - UUID, underscore, length, |, e.g.: '7193287fc88c412c904a1279215fe73a_19'|
+        # - Square bracket array literal with 2 values
+        #   e.g. {{ ['9f6ab9ed86304e8897a87b5b78edd6a1', 'c'] }}
+        #    - UUID
+        #    - Code that appears in templated Jinja block in raw_str
+        for p in output2.split("\0"):
+            if not p:
+                continue
+            # E.g. "2e8577c1d045439ba8d3b9bf47561de3_83"
+            m_id_len = re.match(r"^([0-9a-f]+)_(\d+)$", p)
+            if m_id_len:
+                value = [m_id_len.group(1), int(m_id_len.group(2)), True]
+            else:
+                m_id_start = re.search(r"^([0-9a-f]+)", p)
+                value = [m_id_start.group(0), p[len(m_id_start.group(0)) + 1 :], False]
+            parts.append(value)
+        for alt_id, content_info, literal in parts:
+            old_sliced_file_len = len(self.sliced_file)
+            target_slice_idx = self.find_slice_index(alt_id)
+            if literal:
+                self.move_to_slice(target_slice_idx, content_info)
+            else:
+                self.move_to_slice(target_slice_idx, len(str(content_info)))
+
+            # Sanity check that the template slices we're recording match up
+            # precisely with templated_str.
+            templated_slice_parts = [
+                self.templated_str[tfs.templated_slice]
+                for tfs in self.sliced_file[old_sliced_file_len:]
+            ]
+            templated_slice_str = "".join(templated_slice_parts)
+            if self.sliced_file:
+                sliced_file_str = output1[
+                    self.sliced_file[old_sliced_file_len]
+                    .templated_slice.start : self.sliced_file[-1]
+                    .templated_slice.stop
+                ]
+            else:
+                sliced_file_str = ""
+            if templated_slice_str != sliced_file_str:
+                raise ValueError(
+                    f"Internal error: Templated slice string mismatch: {templated_slice_str} != {sliced_file_str}"
+                )
+
+    def find_slice_index(self, slice_identifier) -> int:
+        """Given a slice identifier (UUID string), return its index."""
+        raw_slices_search_result = [
+            idx
+            for idx, rs in enumerate(self.raw_sliced)
+            if rs.unique_alternate_id == slice_identifier
+        ]
+        if len(raw_slices_search_result) != 1:
+            raise ValueError(
+                f"Internal error. Unable to locate slice for {slice_identifier}."
+            )
+        return raw_slices_search_result[0]
+
+    def move_to_slice(self, target_slice_idx, target_slice_length):
+        """Given a template location, walk execution to that point."""
+        while self.program_counter < len(self.raw_sliced):
+            self.record_trace(
+                target_slice_length if self.program_counter == target_slice_idx else 0
+            )
+            current_raw_slice = self.raw_sliced[self.program_counter]
+            if self.program_counter == target_slice_idx:
+                # Reached the target slice. Go to next location and stop.
+                self.program_counter += 1
+                break
+            elif not current_raw_slice.next_slice_indices:
+                # No choice available. Go to next location.
+                self.program_counter += 1
+            else:
+                # We have choices. Which to choose?
+                candidates = []
+                for next_slice_idx in chain(
+                    current_raw_slice.next_slice_indices, [self.program_counter + 1]
+                ):
+                    if next_slice_idx > target_slice_idx:
+                        # Takes us past the target. No good.
+                        continue
+                    candidates.append(next_slice_idx)
+                candidates.sort(key=lambda c: abs(target_slice_idx - c))
+                self.program_counter = candidates[0]
+        # if self.program_counter == len(self.raw_sliced) and self.program_counter == target_slice_idx:
+        #     # Reached the end of the template without finding target_slice_idx.
+        #     import pdb; pdb.set_trace()
+        #     pass
+
+    def record_trace(self, target_slice_length):
+        """Add the current location to the trace."""
+        slice_type = self.raw_sliced[self.program_counter].slice_type
+        self.sliced_file.append(
+            TemplatedFileSlice(
+                slice_type,
+                slice(
+                    self.raw_sliced[self.program_counter].source_idx,
+                    self.raw_sliced[self.program_counter + 1].source_idx
+                    if self.program_counter + 1 < len(self.raw_sliced)
+                    else len(self.raw_str),
+                ),
+                slice(
+                    self.source_idx,
+                    self.source_idx + target_slice_length
+                    # if slice_type not in ("literal", "templated")
+                    # else self.source_idx + target_slice_length,
+                ),
+            )
+        )
+        if slice_type in ("literal", "templated"):
+            self.source_idx += target_slice_length
 
     @classmethod
-    def _slice_template(cls, in_str: str) -> Iterator[RawFileSlice]:
+    def _slice_template(cls, in_str: str) -> List[RawFileSlice]:
         """Slice template in jinja.
 
         NB: Starts and ends of blocks are not distinguished.
         """
-        env = cls._get_jinja_env()
+        env = JinjaTemplater.get_jinja_env()
         str_buff = ""
         idx = 0
         # We decide the "kind" of element we're dealing with
@@ -330,9 +499,29 @@ def _slice_template(cls, in_str: str) -> Iterator[RawFileSlice]:
         }
 
         # https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.Environment.lex
+        stack = []
+        result = []
+        set_idx = None
         for _, elem_type, raw in env.lex(in_str):
+            # Replace literal text with a unique ID, except for "set"
+            # statements, which don't emit output and thus don't need this
+            # treatment.
             if elem_type == "data":
-                yield RawFileSlice(raw, "literal", idx)
+                if set_idx is None:
+                    unique_alternate_id = uuid.uuid4().hex
+                    alternate_code = f"{unique_alternate_id}_{len(raw)}\0"
+                else:
+                    unique_alternate_id = None
+                    alternate_code = None
+                result.append(
+                    RawFileSlice(
+                        raw,
+                        "literal",
+                        idx,
+                        unique_alternate_id=unique_alternate_id,
+                        alternate_code=alternate_code,
+                    )
+                )
                 idx += len(raw)
                 continue
             str_buff += raw
@@ -373,17 +562,21 @@ def _slice_template(cls, in_str: str) -> Iterator[RawFileSlice]:
                             "Jinja lex() skipped non-whitespace: %s", skipped_str
                         )
                     # Treat the skipped whitespace as a literal.
-                    yield RawFileSlice(skipped_str, "literal", idx)
+                    result.append(
+                        RawFileSlice(skipped_str, "literal", idx, alternate_code="")
+                    )
                     idx += num_chars_skipped
 
             # raw_end and raw_begin behave a little differently in
             # that the whole tag shows up in one go rather than getting
             # parts of the tag at a time.
+            unique_alternate_id = None
+            alternate_code = None
             if elem_type.endswith("_end") or elem_type == "raw_begin":
                 block_type = block_types[elem_type]
                 block_subtype = None
                 # Handle starts and ends of blocks
-                if block_type == "block":
+                if block_type in ("block", "templated"):
                     # Trim off the brackets and then the whitespace
                     m_open = cls.re_open_tag.search(str_buff)
                     m_close = cls.re_close_tag.search(str_buff)
@@ -392,15 +585,39 @@ def _slice_template(cls, in_str: str) -> Iterator[RawFileSlice]:
                         trimmed_content = str_buff[
                             len(m_open.group(0)) : -len(m_close.group(0))
                         ]
-                    if trimmed_content.startswith("end"):
-                        block_type = "block_end"
-                    elif trimmed_content.startswith("el"):
-                        # else, elif
-                        block_type = "block_mid"
+                    if block_type == "block":
+                        if trimmed_content.startswith("end"):
+                            block_type = "block_end"
+                        elif trimmed_content.startswith("el"):
+                            # else, elif
+                            block_type = "block_mid"
+                        else:
+                            block_type = "block_start"
+                            if trimmed_content.split()[0] == "for":
+                                block_subtype = "loop"
                     else:
-                        block_type = "block_start"
-                        if trimmed_content.split()[0] == "for":
-                            block_subtype = "loop"
+                        # For "templated", evaluate the content in case of side
+                        # effects, but return a UUID.
+                        if trimmed_content:
+                            unique_id = uuid.uuid4().hex
+                            unique_alternate_id = unique_id
+                            alternate_code = f"{unique_alternate_id} {m_open.group(1)} {trimmed_content} {m_close.group(1)}\0"
+                if block_type == "block_start" and trimmed_content.split()[0] == "set":
+                    # Jinja supports two forms of {% set %}:
+                    # - {% set variable = value %}
+                    # - {% set variable %}value{% endset %}
+                    # https://jinja.palletsprojects.com/en/2.10.x/templates/#block-assignments
+                    # When the second format is used, leave the value "as is".
+                    # It won't be rendered directly to the template output
+                    # anyway, so substituting our special UUID values would just
+                    # confuse things.
+                    trimmed_content_parts = trimmed_content.split(maxsplit=2)
+                    if len(trimmed_content_parts) <= 2 or not trimmed_content_parts[
+                        2
+                    ].startswith("="):
+                        set_idx = len(result)
+                elif block_type == "block_end" and set_idx is not None:
+                    set_idx = None
                 m = re.search(r"\s+$", raw, re.MULTILINE | re.DOTALL)
                 if raw.startswith("-") and m:
                     # Right whitespace was stripped. Split off the trailing
@@ -411,13 +628,59 @@ def _slice_template(cls, in_str: str) -> Iterator[RawFileSlice]:
                     # returns, it has simply grouped them differently than we
                     # want.
                     trailing_chars = len(m.group(0))
-                    yield RawFileSlice(
-                        str_buff[:-trailing_chars], block_type, idx, block_subtype
+                    result.append(
+                        RawFileSlice(
+                            str_buff[:-trailing_chars],
+                            block_type,
+                            idx,
+                            block_subtype,
+                            unique_alternate_id=unique_alternate_id,
+                            alternate_code=alternate_code,
+                        )
                     )
+                    block_idx = len(result) - 1
                     idx += len(str_buff) - trailing_chars
-                    yield RawFileSlice(str_buff[-trailing_chars:], "literal", idx)
+                    result.append(
+                        RawFileSlice(
+                            str_buff[-trailing_chars:],
+                            "literal",
+                            idx,
+                            alternate_code="",
+                        )
+                    )
                     idx += trailing_chars
                 else:
-                    yield RawFileSlice(str_buff, block_type, idx, block_subtype)
+                    result.append(
+                        RawFileSlice(
+                            str_buff,
+                            block_type,
+                            idx,
+                            block_subtype,
+                            unique_alternate_id=unique_alternate_id,
+                            alternate_code=alternate_code,
+                        )
+                    )
+                    block_idx = len(result) - 1
                     idx += len(str_buff)
+                if block_type == "block_start" and trimmed_content.split()[0] in (
+                    "for",
+                    "if",
+                ):
+                    stack.append(block_idx)
+                elif block_type == "block_mid":
+                    # Record potential forward jump over this block.
+                    result[stack[-1]].next_slice_indices.append(block_idx)
+                    stack.pop()
+                    stack.append(block_idx)
+                elif block_type == "block_end" and trimmed_content.split()[0] in (
+                    "endfor",
+                    "endif",
+                ):
+                    # Record potential forward jump over this block.
+                    result[stack[-1]].next_slice_indices.append(block_idx)
+                    if result[stack[-1]].slice_subtype == "loop":
+                        # Record potential backward jump to the loop beginning.
+                        result[block_idx].next_slice_indices.append(stack[-1] + 1)
+                    stack.pop()
                 str_buff = ""
+        return result
diff --git a/src/sqlfluff/core/templaters/python.py b/src/sqlfluff/core/templaters/python.py
index 01a40b3ca26..df5682cd376 100644
--- a/src/sqlfluff/core/templaters/python.py
+++ b/src/sqlfluff/core/templaters/python.py
@@ -245,10 +245,7 @@ def process(
 
     @classmethod
     def slice_file(
-        cls,
-        raw_str: str,
-        templated_str: str,
-        config=None,
+        cls, raw_str: str, templated_str: str, config=None, **kwargs
     ) -> Tuple[List[RawFileSlice], List[TemplatedFileSlice], str]:
         """Slice the file to determine regions where we can fix."""
         templater_logger.info("Slicing File Template")
@@ -492,31 +489,49 @@ def _split_invariants(
         idx: Optional[int] = None
         templ_idx = 0
         # Loop through
-        for raw, token_type, raw_pos, _ in raw_sliced:
-            if raw in invariants:
+        for raw_file_slice in raw_sliced:
+            if raw_file_slice.raw in invariants:
                 if buffer:
                     yield IntermediateFileSlice(
                         "compound",
-                        slice(idx, raw_pos),
-                        slice(templ_idx, templated_occurrences[raw][0]),
+                        slice(idx, raw_file_slice.source_idx),
+                        slice(templ_idx, templated_occurrences[raw_file_slice.raw][0]),
                         buffer,
                     )
                 buffer = []
                 idx = None
                 yield IntermediateFileSlice(
                     "invariant",
-                    slice(raw_pos, raw_pos + len(raw)),
                     slice(
-                        templated_occurrences[raw][0],
-                        templated_occurrences[raw][0] + len(raw),
+                        raw_file_slice.source_idx,
+                        raw_file_slice.source_idx + len(raw_file_slice.raw),
                     ),
-                    [RawFileSlice(raw, token_type, templated_occurrences[raw][0])],
+                    slice(
+                        templated_occurrences[raw_file_slice.raw][0],
+                        templated_occurrences[raw_file_slice.raw][0]
+                        + len(raw_file_slice.raw),
+                    ),
+                    [
+                        RawFileSlice(
+                            raw_file_slice.raw,
+                            raw_file_slice.slice_type,
+                            templated_occurrences[raw_file_slice.raw][0],
+                        )
+                    ],
+                )
+                templ_idx = templated_occurrences[raw_file_slice.raw][0] + len(
+                    raw_file_slice.raw
                 )
-                templ_idx = templated_occurrences[raw][0] + len(raw)
             else:
-                buffer.append(RawFileSlice(raw, token_type, raw_pos))
+                buffer.append(
+                    RawFileSlice(
+                        raw_file_slice.raw,
+                        raw_file_slice.slice_type,
+                        raw_file_slice.source_idx,
+                    )
+                )
                 if idx is None:
-                    idx = raw_pos
+                    idx = raw_file_slice.source_idx
         # If we have a final buffer, yield it
         if buffer:
             yield IntermediateFileSlice(
diff --git a/src/sqlfluff/rules/L003.py b/src/sqlfluff/rules/L003.py
index 83ff13febe1..4abea97500f 100644
--- a/src/sqlfluff/rules/L003.py
+++ b/src/sqlfluff/rules/L003.py
@@ -147,7 +147,9 @@ def _process_raw_stack(
         templated_file: Optional[TemplatedFile] = None,
     ) -> dict:
         """Take the raw stack, split into lines and evaluate some stats."""
-        raw_stack = cls._reorder_raw_stack(raw_stack, templated_file)
+        # TODO: Is this needed anymore? Perhaps we can roll back most or all of
+        # the work from https://github.com/sqlfluff/sqlfluff/pull/1444.
+        # raw_stack = cls._reorder_raw_stack(raw_stack, templated_file)
         indent_balance = 0
         line_no = 1
         in_indent = True
@@ -274,7 +276,8 @@ def _coerce_indent_to(
             ]
         # Otherwise edit the first element to be the right size
         else:
-            # Edit the first element of this line's indent.
+            # Edit the first element of this line's indent and remove any other
+            # indents.
             fixes = [
                 LintFix(
                     "edit",
@@ -283,7 +286,7 @@ def _coerce_indent_to(
                         raw=desired_indent,
                     ),
                 )
-            ]
+            ] + [LintFix("delete", elem) for elem in current_indent_buffer[1:]]
         return fixes
 
     @staticmethod
@@ -421,13 +424,15 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
             trigger_segment = memory["trigger"]
             if trigger_segment:
                 # Not empty. Process it.
-                result = self._process_current_line(res, memory)
+                result = self._process_current_line(res, memory, context)
                 if context.segment.is_type("newline"):
                     memory["trigger"] = None
                 return result
         return LintResult(memory=memory)
 
-    def _process_current_line(self, res: dict, memory: dict) -> LintResult:
+    def _process_current_line(
+        self, res: dict, memory: dict, context: RuleContext
+    ) -> LintResult:
         """Checks indentation of one line of code, returning a LintResult.
 
         The _eval() function calls it for the current line of code:
@@ -507,6 +512,84 @@ def _process_current_line(self, res: dict, memory: dict) -> LintResult:
                     ],
                 )
 
+        # Special handling for template end blocks on a line by themselves.
+        if self._is_template_block_end_line(
+            this_line["line_buffer"], context.templated_file
+        ):
+            block_lines = {
+                k: (
+                    "end"
+                    if self._is_template_block_end_line(
+                        res[k]["line_buffer"], context.templated_file
+                    )
+                    else "start",
+                    res[k]["indent_balance"],
+                    "".join(
+                        seg.raw or getattr(seg, "source_str", "")
+                        for seg in res[k]["line_buffer"]
+                    ),
+                )
+                for k in res
+                if self._is_template_block_end_line(
+                    res[k]["line_buffer"], context.templated_file
+                )
+                or self._is_template_block_start_line(
+                    res[k]["line_buffer"], context.templated_file
+                )
+            }
+
+            # For a template block end on a line by itself, search for a
+            # matching block start on a line by itself. If there is one, match
+            # its indentation. Question: Could we avoid treating this as a
+            # special case? It has some similarities to the non-templated test
+            # case test/fixtures/linter/indentation_error_contained.sql, in tha
+            # both have lines where indent_balance drops 2 levels from one line
+            # to the next, making it a bit unclear how to indent that line.
+            template_block_level = -1
+            for k in sorted(block_lines.keys(), reverse=True):
+                if block_lines[k][0] == "end":
+                    template_block_level -= 1
+                else:
+                    template_block_level += 1
+
+                if template_block_level != 0:
+                    continue
+
+                # Found prior template block line with the same indent balance.
+
+                # Is this a problem line?
+                if k in memory["problem_lines"] + memory["hanging_lines"]:
+                    # Skip it if it is
+                    return LintResult(memory=memory)
+
+                self.logger.debug("    [template block end] Comparing to #%s", k)
+                if this_line["indent_size"] == res[k]["indent_size"]:
+                    # All good.
+                    return LintResult(memory=memory)
+
+                # Indents don't match even though balance is the same...
+                memory["problem_lines"].append(this_line_no)
+
+                # The previous indent.
+                desired_indent = "".join(elem.raw for elem in res[k]["indent_buffer"])
+
+                # Make fixes
+                fixes = self._coerce_indent_to(
+                    desired_indent=desired_indent,
+                    current_indent_buffer=this_line["indent_buffer"],
+                    current_anchor=this_line["line_buffer"][0],
+                )
+                self.logger.debug(
+                    "    !! Indentation does not match #%s. Fixes: %s", k, fixes
+                )
+                return LintResult(
+                    anchor=trigger_segment,
+                    memory=memory,
+                    description="Indentation not consistent with line #{}".format(k),
+                    # See above for logic
+                    fixes=fixes,
+                )
+
         # Assuming it's not a hanger, let's compare it to the other previous
         # lines. We do it in reverse so that closer lines are more relevant.
         for k in sorted(res.keys(), reverse=True):
@@ -517,7 +600,10 @@ def _process_current_line(self, res: dict, memory: dict) -> LintResult:
                 continue
 
             # Is this an empty line?
-            if not any(elem.is_code for elem in res[k]["line_buffer"]):
+            if not any(
+                elem.is_code or elem.is_type("placeholder")
+                for elem in res[k]["line_buffer"]
+            ):
                 # Skip if it is
                 continue
 
@@ -526,8 +612,9 @@ def _process_current_line(self, res: dict, memory: dict) -> LintResult:
             # If we're comparing to a previous, more deeply indented line, then skip and keep looking.
             if indent_diff < 0:
                 continue
+
             # Is the indent balance the same?
-            elif indent_diff == 0:
+            if indent_diff == 0:
                 self.logger.debug("    [same indent balance] Comparing to #%s", k)
                 if this_line["indent_size"] != res[k]["indent_size"]:
                     # Indents don't match even though balance is the same...
@@ -797,3 +884,53 @@ def _get_element_template_info(
             if slices:
                 return slices[0].slice_type
         return None
+
+    @classmethod
+    def _single_placeholder_line(cls, current_line):
+        count_placeholder = 0
+        for seg in current_line:
+            if seg.is_code:
+                return False
+            elif seg.is_type("placeholder"):
+                count_placeholder += 1
+        return count_placeholder == 1
+
+    @classmethod
+    def _is_template_block_start_line(cls, current_line, templated_file):
+        def segment_info(idx: int) -> Tuple[str, Optional[str]]:
+            """Helper function."""
+            seg = current_line[idx]
+            return seg.type, cls._get_element_template_info(seg, templated_file)
+
+        if not cls._single_placeholder_line(current_line):
+            return False
+        for idx in range(1, len(current_line)):
+            if (
+                segment_info(idx - 1)
+                in (
+                    ("placeholder", "block_start"),
+                    ("placeholder", "compound"),
+                    ("placeholder", "block_mid"),
+                )
+                and segment_info(idx) == ("indent", None)
+            ):
+                return True
+        return False
+
+    @classmethod
+    def _is_template_block_end_line(cls, current_line, templated_file):
+        def segment_info(idx: int) -> Tuple[str, Optional[str]]:
+            """Helper function."""
+            seg = current_line[idx]
+            return seg.type, cls._get_element_template_info(seg, templated_file)
+
+        if not cls._single_placeholder_line(current_line):
+            return False
+        for idx in range(1, len(current_line)):
+            if segment_info(idx - 1) == ("dedent", None) and segment_info(idx) in (
+                ("placeholder", "block_end"),
+                ("placeholder", "compound"),
+                ("placeholder", "block_mid"),
+            ):
+                return True
+        return False
diff --git a/test/core/templaters/jinja_test.py b/test/core/templaters/jinja_test.py
index 047aaac4983..baf97488d06 100644
--- a/test/core/templaters/jinja_test.py
+++ b/test/core/templaters/jinja_test.py
@@ -6,6 +6,7 @@
 import pytest
 
 from sqlfluff.core.templaters import JinjaTemplater
+from sqlfluff.core.templaters.jinja import TemplateTracer
 from sqlfluff.core import Linter, FluffConfig
 
 
@@ -86,10 +87,12 @@ class RawTemplatedTestCase(NamedTuple):
             instr="\n\n{%- set x = 42 %}\nSELECT 1, 2\n",
             templated_str="\nSELECT 1, 2\n",
             expected_templated_sliced__source_list=[
-                "\n\n{%- set x = 42 %}",
+                "\n\n",
+                "{%- set x = 42 %}",
                 "\nSELECT 1, 2\n",
             ],
             expected_templated_sliced__templated_list=[
+                "",
                 "",
                 "\nSELECT 1, 2\n",
             ],
@@ -104,10 +107,14 @@ class RawTemplatedTestCase(NamedTuple):
             instr="\n\n{%- set x = 42 -%}\nSELECT 1, 2\n",
             templated_str="SELECT 1, 2\n",
             expected_templated_sliced__source_list=[
-                "\n\n{%- set x = 42 -%}\n",
+                "\n\n",
+                "{%- set x = 42 -%}",
+                "\n",
                 "SELECT 1, 2\n",
             ],
             expected_templated_sliced__templated_list=[
+                "",
+                "",
                 "",
                 "SELECT 1, 2\n",
             ],
@@ -157,12 +164,14 @@ class RawTemplatedTestCase(NamedTuple):
 """,
             expected_templated_sliced__source_list=[
                 "SELECT\n  ",
-                "{{ 'col1,' -}}\n  ",
+                "{{ 'col1,' -}}",
+                "\n  ",
                 "col2\n",
             ],
             expected_templated_sliced__templated_list=[
                 "SELECT\n  ",
                 "col1,",
+                "",
                 "col2\n",
             ],
             expected_raw_sliced__source_list=[
@@ -184,12 +193,16 @@ class RawTemplatedTestCase(NamedTuple):
 """,
             expected_templated_sliced__source_list=[
                 "select\n    c1,",
-                "\n    {{- 'c' -}}\n",
+                "\n    ",
+                "{{- 'c' -}}",
+                "\n",
                 "2 as user_id\n",
             ],
             expected_templated_sliced__templated_list=[
                 "select\n    c1,",
+                "",
                 "c",
+                "",
                 "2 as user_id\n",
             ],
             expected_raw_sliced__source_list=[
@@ -211,12 +224,16 @@ class RawTemplatedTestCase(NamedTuple):
 """,
             expected_templated_sliced__source_list=[
                 "select\n    c1,",
-                "\n    {#- Column 2 -#} ",
+                "\n    ",
+                "{#- Column 2 -#}",
+                " ",
                 "c2 as user_id\n",
             ],
             expected_templated_sliced__templated_list=[
                 "select\n    c1,",
                 "",
+                "",
+                "",
                 "c2 as user_id\n",
             ],
             expected_raw_sliced__source_list=[
@@ -227,6 +244,85 @@ class RawTemplatedTestCase(NamedTuple):
                 "c2 as user_id\n",
             ],
         ),
+        RawTemplatedTestCase(
+            name="union_all_loop1",
+            instr="""{% set products = [
+  'table1',
+  'table2',
+  ] %}
+
+{% for product in products %}
+SELECT
+  brand
+FROM
+  {{ product }}
+{% if not loop.last -%} UNION ALL {%- endif %}
+{% endfor %}
+""",
+            templated_str="\n\n\nSELECT\n  brand\nFROM\n  table1\nUNION ALL\n\nSELECT\n  brand\nFROM\n  table2\n\n\n",
+            expected_templated_sliced__source_list=[
+                "{% set products = [\n  'table1',\n  'table2',\n  ] %}",
+                "\n\n",
+                "{% for product in products %}",
+                "\nSELECT\n  brand\nFROM\n  ",
+                "{{ product }}",
+                "\n",
+                "{% if not loop.last -%}",
+                " ",
+                "UNION ALL",
+                " ",
+                "{%- endif %}",
+                "\n",
+                "{% endfor %}",
+                "\nSELECT\n  brand\nFROM\n  ",
+                "{{ product }}",
+                "\n",
+                "{% if not loop.last -%}",
+                "{%- endif %}",
+                "\n",
+                "{% endfor %}",
+                "\n",
+            ],
+            expected_templated_sliced__templated_list=[
+                "",
+                "\n\n",
+                "",
+                "\nSELECT\n  brand\nFROM\n  ",
+                "table1",
+                "\n",
+                "",
+                "",
+                "UNION ALL",
+                "",
+                "",
+                "\n",
+                "",
+                "\nSELECT\n  brand\nFROM\n  ",
+                "table2",
+                "\n",
+                "",
+                "",
+                "\n",
+                "",
+                "\n",
+            ],
+            expected_raw_sliced__source_list=[
+                "{% set products = [\n  'table1',\n  'table2',\n  ] %}",
+                "\n\n",
+                "{% for product in products %}",
+                "\nSELECT\n  brand\nFROM\n  ",
+                "{{ product }}",
+                "\n",
+                "{% if not loop.last -%}",
+                " ",
+                "UNION ALL",
+                " ",
+                "{%- endif %}",
+                "\n",
+                "{% endfor %}",
+                "\n",
+            ],
+        ),
     ],
     ids=lambda case: case.name,
 )
@@ -413,28 +509,32 @@ def test__templater_full(subpath, code_only, include_meta, yaml_loader, caplog):
 )
 def test__templater_jinja_slice_template(test, result):
     """Test _slice_template."""
-    resp = list(JinjaTemplater._slice_template(test))
+    resp = list(TemplateTracer._slice_template(test))
     # check contigious (unless there's a comment in it)
     if "{#" not in test:
-        assert "".join(elem[0] for elem in resp) == test
+        assert "".join(elem.raw for elem in resp) == test
         # check indices
         idx = 0
-        for literal, _, pos, _ in resp:
-            assert pos == idx
-            idx += len(literal)
+        for raw_slice in resp:
+            assert raw_slice.source_idx == idx
+            idx += len(raw_slice.raw)
     # Check total result
-    assert [r[:3] for r in resp] == result
+    assert [
+        (raw_slice.raw, raw_slice.slice_type, raw_slice.source_idx)
+        for raw_slice in resp
+    ] == result
 
 
 @pytest.mark.parametrize(
-    "raw_file,templated_file,result",
+    "raw_file,templated_file,override_context,result",
     [
-        ("", "", []),
-        ("foo", "foo", [("literal", slice(0, 3, None), slice(0, 3, None))]),
+        ("", "", None, []),
+        ("foo", "foo", None, [("literal", slice(0, 3, None), slice(0, 3, None))]),
         # Example with no loops
         (
             "SELECT {{blah}}, boo {# comment #} from something",
             "SELECT foobar, boo  from something",
+            dict(blah="foobar"),
             [
                 ("literal", slice(0, 7, None), slice(0, 7, None)),
                 ("templated", slice(7, 15, None), slice(7, 13, None)),
@@ -447,6 +547,7 @@ def test__templater_jinja_slice_template(test, result):
         (
             "SELECT {# A comment #} {{field}} {% for i in [1, 3, 7]%}, fld_{{i}}_x{% endfor %} FROM my_schema.{{my_table}} ",
             "SELECT  foobar , fld_1_x, fld_3_x, fld_7_x FROM my_schema.barfoo ",
+            dict(field="foobar", my_table="barfoo"),
             [
                 ("literal", slice(0, 7, None), slice(0, 7, None)),
                 ("comment", slice(7, 22, None), slice(7, 7, None)),
@@ -457,9 +558,11 @@ def test__templater_jinja_slice_template(test, result):
                 ("literal", slice(56, 62, None), slice(15, 21, None)),
                 ("templated", slice(62, 67, None), slice(21, 22, None)),
                 ("literal", slice(67, 69, None), slice(22, 24, None)),
+                ("block_end", slice(69, 81, None), slice(24, 24, None)),
                 ("literal", slice(56, 62, None), slice(24, 30, None)),
                 ("templated", slice(62, 67, None), slice(30, 31, None)),
                 ("literal", slice(67, 69, None), slice(31, 33, None)),
+                ("block_end", slice(69, 81, None), slice(33, 33, None)),
                 ("literal", slice(56, 62, None), slice(33, 39, None)),
                 ("templated", slice(62, 67, None), slice(39, 40, None)),
                 ("literal", slice(67, 69, None), slice(40, 42, None)),
@@ -473,6 +576,7 @@ def test__templater_jinja_slice_template(test, result):
         (
             "SELECT {# A comment #} {{field}} {% for i in [1, 3, 7]%}, fld_{{i}}{% endfor %} FROM my_schema.{{my_table}} ",
             "SELECT  foobar , fld_1, fld_3, fld_7 FROM my_schema.barfoo ",
+            dict(field="foobar", my_table="barfoo"),
             [
                 ("literal", slice(0, 7, None), slice(0, 7, None)),
                 ("comment", slice(7, 22, None), slice(7, 7, None)),
@@ -482,8 +586,10 @@ def test__templater_jinja_slice_template(test, result):
                 ("block_start", slice(33, 56, None), slice(15, 15, None)),
                 ("literal", slice(56, 62, None), slice(15, 21, None)),
                 ("templated", slice(62, 67, None), slice(21, 22, None)),
+                ("block_end", slice(67, 79, None), slice(22, 22, None)),
                 ("literal", slice(56, 62, None), slice(22, 28, None)),
                 ("templated", slice(62, 67, None), slice(28, 29, None)),
+                ("block_end", slice(67, 79, None), slice(29, 29, None)),
                 ("literal", slice(56, 62, None), slice(29, 35, None)),
                 ("templated", slice(62, 67, None), slice(35, 36, None)),
                 ("block_end", slice(67, 79, None), slice(36, 36, None)),
@@ -496,6 +602,10 @@ def test__templater_jinja_slice_template(test, result):
         (
             "{{ config(materialized='view') }}\n\nSELECT 1 FROM {{ source('finance', 'reconciled_cash_facts') }}\n\n",
             "\n\nSELECT 1 FROM finance_reconciled_cash_facts\n\n",
+            dict(
+                config=lambda *args, **kwargs: "",
+                source=lambda *args, **kwargs: "finance_reconciled_cash_facts",
+            ),
             [
                 ("templated", slice(0, 33, None), slice(0, 0, None)),
                 ("literal", slice(33, 49, None), slice(0, 16, None)),
@@ -515,21 +625,22 @@ def test__templater_jinja_slice_template(test, result):
             "c_2+42 AS the_meaning_of_liff\n    \n        , "
             "c_3+42 AS the_meaning_of_lifff\n    \n"
             "FROM my_table",
+            None,
             [
                 ("literal", slice(0, 11, None), slice(0, 11, None)),
                 ("block_start", slice(11, 35, None), slice(11, 11, None)),
                 ("literal", slice(35, 48, None), slice(11, 24, None)),
                 ("templated", slice(48, 53, None), slice(24, 25, None)),
                 ("literal", slice(53, 77, None), slice(25, 49, None)),
-                # NB: A templated section which loops back, spans the whole section.
-                # We get to match it more accurately here because we're lucky.
-                ("templated", slice(77, 95, None), slice(49, 55, None)),
+                ("templated", slice(77, 90, None), slice(49, 50, None)),
+                ("literal", slice(90, 95, None), slice(50, 55, None)),
+                ("block_end", slice(95, 107, None), slice(55, 55, None)),
                 ("literal", slice(35, 48, None), slice(55, 68, None)),
                 ("templated", slice(48, 53, None), slice(68, 69, None)),
                 ("literal", slice(53, 77, None), slice(69, 93, None)),
-                # NB: A templated section which loops back, spans the whole section.
-                # We get to match it more accurately here because we're lucky.
-                ("templated", slice(77, 95, None), slice(93, 100, None)),
+                ("templated", slice(77, 90, None), slice(93, 95, None)),
+                ("literal", slice(90, 95, None), slice(95, 100, None)),
+                ("block_end", slice(95, 107, None), slice(100, 100, None)),
                 ("literal", slice(35, 48, None), slice(100, 113, None)),
                 ("templated", slice(48, 53, None), slice(113, 114, None)),
                 ("literal", slice(53, 77, None), slice(114, 138, None)),
@@ -539,38 +650,37 @@ def test__templater_jinja_slice_template(test, result):
                 ("literal", slice(107, 121, None), slice(146, 160, None)),
             ],
         ),
-        # Test a wrapped example. Given the default config is to unwrap any wrapped
-        # queries, it should ignore the ends in the sliced file.
-        (
-            "SELECT {{blah}} FROM something",
-            "WITH wrap AS (SELECT nothing FROM something) SELECT * FROM wrap",
-            # The sliced version should have trimmed the ends
-            [
-                ("literal", slice(0, 7, None), slice(0, 7, None)),
-                ("templated", slice(7, 15, None), slice(7, 14, None)),
-                ("literal", slice(15, 30, None), slice(14, 29, None)),
-            ],
-        ),
         # Test an example where a block is removed entirely.
         (
             "{% set thing %}FOO{% endset %} SELECT 1",
             " SELECT 1",
-            # There should be a zero length templated part at the start.
+            None,
             [
-                # The templated section at the start should be entirely
-                # templated and not include a distinct literal within it.
-                ("templated", slice(0, 30, None), slice(0, 0, None)),
+                ("block_start", slice(0, 15, None), slice(0, 0, None)),
+                ("literal", slice(15, 18, None), slice(0, 0, None)),
+                ("block_end", slice(18, 30, None), slice(0, 0, None)),
                 ("literal", slice(30, 39, None), slice(0, 9, None)),
             ],
         ),
     ],
 )
-def test__templater_jinja_slice_file(raw_file, templated_file, result, caplog):
+def test__templater_jinja_slice_file(
+    raw_file, templated_file, override_context, result, caplog
+):
     """Test slice_file."""
+    templater = JinjaTemplater(override_context=override_context)
+    env = templater.get_jinja_env()
+    live_context = templater.get_context()
+
+    def make_template(in_str):
+        return env.from_string(in_str, globals=live_context)
+
+    # TODO: Now that we're generating "templated_file", eliminate this field
+    # from the parametrized test cases.
+    templated_file = make_template(raw_file).render()
     with caplog.at_level(logging.DEBUG, logger="sqlfluff.templater"):
         _, resp, _ = JinjaTemplater.slice_file(
-            raw_file,
-            templated_file,
+            raw_file, templated_file, make_template=make_template
         )
     # Check contigious on the TEMPLATED VERSION
     print(resp)
@@ -585,4 +695,12 @@ def test__templater_jinja_slice_file(raw_file, templated_file, result, caplog):
         if elem[0] == "literal":
             assert elem[1] is not None
     # check result
-    assert resp == result
+    actual = [
+        (
+            templated_file_slice.slice_type,
+            templated_file_slice.source_slice,
+            templated_file_slice.templated_slice,
+        )
+        for templated_file_slice in resp
+    ]
+    assert actual == result
diff --git a/test/core/templaters/python_test.py b/test/core/templaters/python_test.py
index 36c5ddd1f0d..f16a404adb6 100644
--- a/test/core/templaters/python_test.py
+++ b/test/core/templaters/python_test.py
@@ -192,12 +192,12 @@ def test__templater_python_slice_template(test, result):
     """Test _slice_template."""
     resp = list(PythonTemplater._slice_template(test))
     # check contigious
-    assert "".join(elem[0] for elem in resp) == test
+    assert "".join(elem.raw for elem in resp) == test
     # check indices
     idx = 0
-    for literal, _, pos, _ in resp:
-        assert pos == idx
-        idx += len(literal)
+    for raw_file_slice in resp:
+        assert raw_file_slice.source_idx == idx
+        idx += len(raw_file_slice.raw)
     # Check total result
     assert resp == result
 
diff --git a/test/fixtures/linter/autofix/bigquery/003_templating/after.sql b/test/fixtures/linter/autofix/bigquery/003_templating/after.sql
index af1781d534f..a9278b024ba 100644
--- a/test/fixtures/linter/autofix/bigquery/003_templating/after.sql
+++ b/test/fixtures/linter/autofix/bigquery/003_templating/after.sql
@@ -18,11 +18,11 @@ FROM
     {% if loop.first %}
         -- This next section gets very strange with forced template_blocks_indent
         {{action}}_raw_effect_sizes
-        {% else %}
-        JOIN
-                    {{action}}_raw_effect_sizes
-            USING
-                            ({{corr_states}})
-{% endif %}
+    {% else %}
+    JOIN
+            {{action}}_raw_effect_sizes
+        USING
+                    ({{corr_states}})
+    {% endif %}
 {% endfor %}
 CROSS JOIN action_states
diff --git a/test/fixtures/rules/std_rule_cases/L003.yml b/test/fixtures/rules/std_rule_cases/L003.yml
index 54c051bf79b..0778ded86ca 100644
--- a/test/fixtures/rules/std_rule_cases/L003.yml
+++ b/test/fixtures/rules/std_rule_cases/L003.yml
@@ -339,14 +339,14 @@ test_jinja_indent_templated_table_name_a:
   fix_str: |
     -- This file combines product data from individual brands into a staging table
     {% for product in ['table1', 'table2'] %}
-    SELECT
-        brand,
-        country_code,
-        category,
-        name,
-        id
-    FROM
-        {{ product }}
+        SELECT
+            brand,
+            country_code,
+            category,
+            name,
+            id
+        FROM
+            {{ product }}
     {% if not loop.last -%} UNION ALL {%- endif %}
     {% endfor %}
 
@@ -369,17 +369,42 @@ test_jinja_indent_templated_table_name_b:
   fix_str: |
     -- This file combines product data from individual brands into a staging table
     {% for product in ['table1', 'table2'] %}
-    SELECT
-        brand,
-        country_code,
-        category,
-        name,
-        id
-    FROM
-        {{ product }}
+        SELECT
+            brand,
+            country_code,
+            category,
+            name,
+            id
+        FROM
+            {{ product }}
     {% if not loop.last -%} UNION ALL {%- endif %}
     {% endfor %}
 
+test_jinja_nested_blocks:
+  fail_str: |
+    WITH
+    raw_effect_sizes AS (
+    SELECT
+    {% for action in ['a'] %}
+    {% if True %}
+    rate_su_{{action}},
+    {% endif %}
+    {% endfor %}
+    )
+    SELECT 1
+
+  fix_str: |
+    WITH
+    raw_effect_sizes AS (
+        SELECT
+            {% for action in ['a'] %}
+                {% if True %}
+                    rate_su_{{action}},
+                {% endif %}
+            {% endfor %}
+    )
+    SELECT 1
+
 # LIMIT and QUALIFY both indent
 test_limit_and_qualify_indent:
   fail_str: |
diff --git a/test/fixtures/templater/jinja_l_metas/001.yml b/test/fixtures/templater/jinja_l_metas/001.yml
index 767d994977a..205a6509f1c 100644
--- a/test/fixtures/templater/jinja_l_metas/001.yml
+++ b/test/fixtures/templater/jinja_l_metas/001.yml
@@ -1,5 +1,5 @@
 file:
-  - placeholder: '{% set some_condition %}TRUE{% endset %}'
+  - placeholder: '{% set some_condition %}...{% endset %}'
   - newline: "\n"
   - newline: "\n"
   - statement:
@@ -77,4 +77,4 @@ file:
                 table_expression:
                   table_reference:
                     identifier: cust
-              dedent: ''
\ No newline at end of file
+              dedent: ''
diff --git a/test/fixtures/templater/jinja_l_metas/002.yml b/test/fixtures/templater/jinja_l_metas/002.yml
index 397f57ecd63..6dc43c84c59 100644
--- a/test/fixtures/templater/jinja_l_metas/002.yml
+++ b/test/fixtures/templater/jinja_l_metas/002.yml
@@ -92,12 +92,13 @@ file:
         - newline: "\n"
         - whitespace: '        '
         - dedent: ''
-        - placeholder: '{% else %}'
-        - indent: ''
+        - placeholder: '{% else %}... [103 omitted] ...{% endif %}'
         - newline: "\n"
         - whitespace: '    '
         - newline: "\n"
         - whitespace: '        '
+        - placeholder: '{% if loop.first %}... [49 omitted] ...{% else %}'
+        - indent: ''
         - newline: "\n"
         - whitespace: '        '
         - dedent: ''
@@ -130,11 +131,14 @@ file:
           - dedent: ''
         - newline: "\n"
         - whitespace: '        '
-        - placeholder: '{% if loop.first %}{% else %}{% endif %}'
+        - dedent: ''
+        - placeholder: '{% endif %}'
         - newline: "\n"
         - whitespace: '    '
         - newline: "\n"
         - whitespace: '        '
+        - placeholder: '{% if loop.first %}... [49 omitted] ...{% else %}'
+        - indent: ''
         - newline: "\n"
         - whitespace: '        '
         - join_clause:


Code review comments:
diff_hunk: @@ -411,13 +628,59 @@ def _slice_template(cls, in_str: str) -> Iterator[RawFileSlice]:
                     # returns, it has simply grouped them differently than we
                     # want.
                     trailing_chars = len(m.group(0))
-                    yield RawFileSlice(
-                        str_buff[:-trailing_chars], block_type, idx, block_subtype
+                    result.append(
+                        RawFileSlice(
+                            str_buff[:-trailing_chars],
+                            block_type,
+                            idx,
+                            block_subtype,
+                            unique_alternate_id=unique_alternate_id,
+                            alternate_code=alternate_code,
+                        )
                     )
+                    block_idx = len(result) - 1
path: src/sqlfluff/core/templaters/jinja.py
text: @user1:
Is constructing `block_idx` like this reliable enough? This feels potentially risky. At the very least we should document why we're constructing it like this, or perhaps track it differently? We could even explicitly keep a reference to the last `start_block` as a way of being able to refer back to it.

diff_hunk: @@ -23,13 +24,17 @@ def iter_indices_of_newlines(raw_str: str) -> Iterator[int]:
             break  # pragma: no cover TODO?
 
 
-class RawFileSlice(NamedTuple):
+@dataclass
+class RawFileSlice:
     """A slice referring to a raw file."""
 
     raw: str
     slice_type: str
     source_idx: int
     slice_subtype: Optional[str] = None
+    unique_alternate_id: Optional[str] = None
+    alternate_code: Optional[str] = None
path: src/sqlfluff/core/templaters/base.py
text: @user1:
If these are actually UUIDs, then we should probably use the UUID type hints: https://docs.python.org/3/library/uuid.html

@author:
They only need to be unique within a template, so they wouldn't have to be UUIDs. For example, an integer counter would have also worked. I'm storing them as strings because they become part of the (Jinja) template and then its output.

diff_hunk: @@ -401,6 +401,51 @@ def elements_to_segments(
                 # only one of them.
                 if not placeholder_str:
                     placeholder_str = "".join(s.raw for s in so_slices)
+                # The Jinja templater sometimes returns source-only slices with
+                # gaps between. For example, in this section:
+                #
+                #   {% else %}
+                #   JOIN
+                #       {{action}}_raw_effect_sizes
+                #   USING
+                #       ({{ states }})
+                #   {% endif %}
+                #
+                # , we might get {% else %} and {% endif %} slices, without the
+                # 4 lines between. This indicates those lines were not executed
+                # In this case, generate a placeholder where the skipped code is
+                # omitted but noted with a brief string, e.g.:
+                #
+                # "{% else %}... [103 omitted] ...{% endif %}".
path: src/sqlfluff/core/parser/lexer.py
text: @user1:
Doing this makes a lot of sense, but I don't think that it's clear from `103 ommitted` what the significance is of the omission. I think something more explicit like `103 unused template characters` might be clearer?

@author:
I struggled to find the right wording here. These are parts of the template that were literally skipped over during execution. "Unused" does seem clearer than "omitted".

diff_hunk: @@ -401,6 +401,51 @@ def elements_to_segments(
                 # only one of them.
                 if not placeholder_str:
                     placeholder_str = "".join(s.raw for s in so_slices)
+                # The Jinja templater sometimes returns source-only slices with
+                # gaps between. For example, in this section:
+                #
+                #   {% else %}
+                #   JOIN
+                #       {{action}}_raw_effect_sizes
+                #   USING
+                #       ({{ states }})
+                #   {% endif %}
+                #
+                # , we might get {% else %} and {% endif %} slices, without the
+                # 4 lines between. This indicates those lines were not executed
+                # In this case, generate a placeholder where the skipped code is
+                # omitted but noted with a brief string, e.g.:
+                #
+                # "{% else %}... [103 omitted] ...{% endif %}".
+                #
+                # This is more readable -- it would be REALLY confusing for a
+                # placeholder to include code that wasn't even executed!!
+                if len(so_slices) >= 2:
+                    has_gap = False
+                    gap_placeholder_parts = []
+                    last_slice = None
+                    # For each slice...
+                    for so_slice in so_slices:
+                        # If it's not the first slice, was there a gap?
+                        if last_slice:
+                            end_last = last_slice.source_idx + len(last_slice.raw)
+                            num_omitted = so_slice.source_idx - end_last
path: src/sqlfluff/core/parser/lexer.py
text: @user1:
`chars_omitted`? Might be more explanatory for what the number means.

@author:
How about `chars_skipped`?

diff_hunk: @@ -15,6 +15,10 @@ class MetaSegment(RawSegment):
     indent_val = 0
     is_meta = True
 
+    def __init__(self, template=False, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.template = template
path: src/sqlfluff/core/parser/segments/meta.py
text: @user1:
I think `template` here should probably be `is_template` to be in line with the other similar flags.