Repository: sqlfluff/sqlfluff

Issue:
Fatal templating error with Jinja templater. Tracer produces odd results.
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### What Happened

Issue found while assessing an Airflow project.

The smallest query I can make which triggers the issue is: 
```sql
SELECT
	{% block table_name %}a{% endblock %}.b
FROM d.{{ self.table_name() }}
```

When running this query through `lint` I get an `AssertionError`, or if running on the more friendly error message PR (#3433) I get: `WARNING    Length of templated file mismatch with final slice: 21 != 19.`.

### Expected Behaviour

This query should slice properly and probably eventually give a jinja error that the required variables are undefined.

### Observed Behaviour

I've dug a little into the error and the sliced file being produced is:

```python
[
    TemplatedFileSlice(slice_type='literal', source_slice=slice(0, 8, None), templated_slice=slice(0, 8, None)),
    TemplatedFileSlice(slice_type='block_start', source_slice=slice(8, 30, None), templated_slice=slice(8, 8, None)),
    TemplatedFileSlice(slice_type='literal', source_slice=slice(30, 31, None), templated_slice=slice(8, 9, None)),
    TemplatedFileSlice(slice_type='block_end', source_slice=slice(31, 45, None), templated_slice=slice(9, 9, None)),
    TemplatedFileSlice(slice_type='literal', source_slice=slice(45, 55, None), templated_slice=slice(9, 19, None)),
    TemplatedFileSlice(slice_type='templated', source_slice=slice(55, 78, None), templated_slice=slice(19, 19, None)),
    TemplatedFileSlice(slice_type='literal', source_slice=slice(78, 79, None), templated_slice=slice(19, 19, None))
]
```

The issue is that while the `source_slice` looks correct for the slices, almost all of the `templated_slices` values have zero length, and importantly the last one doesn't end at position 21.

The rendered file is `SELECT\n\ta.b\nFROM d.a\n` (I've included the escape chars) which is indeed 21 chars long.

@barrywhart I might need your help to work out what's going on with the Jinja tracer here.

### How to reproduce

Run provided query, `main` branch. Set to the `jinja` templater.

### Dialect

dialect is set to `snowflake`, but I don't think we're getting far enough for that to make a difference.

### Version

`main` branch commit `cb6357c540d2d968f766f3a7a4fa16f231cb80e4` (and a few branches derived from it)

### Configuration

N/A

### Are you willing to work on and submit a PR to address the issue?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)



Discussion:
I'll take a look.

And darn it -- first bug report against this code in the past couple months, I think. 😅
Starting to look at this. One problem I noticed (perhaps not the only one) is that the trailing literal newline in the source string has no corresponding templated slice, so it's like building the templated slice array has stopped early for some reason.

The 0-length slices may be legit. Will share more as I learn things, but is `{% block %}` a Jinja builtin or an extension? If it's an extension, maybe base Jinja is just skipping it (i.e. rendering it as empty string). 
Ok, I think the issue is not related to undefined variables. I get the same assertion error if I define the variable prior to the block, e.g.:
```
{% set table_name = "abc" %}
SELECT {% block table_name %}a{% endblock %} FROM {{ self.table_name() }}
```

I'm pretty sure the real issue is that we aren't handling `{% block %}` correctly **at all** (probably because I hadn't heard of it before 🤪).

II think it should be handled similarly to `{% set %}` or `{% macro %}` blocks, i.e. basically don't trace when they are **defined**, only when they are **used**.

I should be able to fix it this week. For now, just need to let my brain recover from looking at this code again. Even though I wrote it, it's a little too "meta" for me to stare at it for more than 1-2 hours at a time. 😅

Patch:
diff --git a/src/sqlfluff/core/templaters/slicers/tracer.py b/src/sqlfluff/core/templaters/slicers/tracer.py
--- a/src/sqlfluff/core/templaters/slicers/tracer.py
+++ b/src/sqlfluff/core/templaters/slicers/tracer.py
@@ -40,6 +40,7 @@ class RawSliceInfo:
     unique_alternate_id: Optional[str]
     alternate_code: Optional[str]
     next_slice_indices: List[int] = field(default_factory=list)
+    inside_block: bool = field(default=False)  # {% block %}
 
 
 class JinjaTracer:
@@ -101,13 +102,21 @@ def trace(self, append_to_templated: str = "") -> JinjaTrace:
             alt_id, content_info, literal = value
             target_slice_idx = self.find_slice_index(alt_id)
             slice_length = content_info if literal else len(str(content_info))
-            self.move_to_slice(target_slice_idx, slice_length)
+            target_inside_block = self.raw_slice_info[
+                self.raw_sliced[target_slice_idx]
+            ].inside_block
+            if not target_inside_block:
+                # Normal case: Walk through the template.
+                self.move_to_slice(target_slice_idx, slice_length)
+            else:
+                # {% block %} executes code elsewhere in the template but does
+                # not move there. It's a bit like macro invocation.
+                self.record_trace(slice_length, target_slice_idx)
 
         # TRICKY: The 'append_to_templated' parameter is only used by the dbt
         # templater, passing "\n" for this parameter if we need to add one back.
         # (The Jinja templater does not pass this parameter, so
         # 'append_to_templated' gets the default value of "", empty string.)
-        # we receive the default value of "".) The dbt templater will
         # For more detail, see the comments near the call to slice_file() in
         # plugins/sqlfluff-templater-dbt/sqlfluff_templater_dbt/templater.py.
         templated_str = self.make_template(self.raw_str).render() + append_to_templated
@@ -197,7 +206,8 @@ def __init__(self, raw_str: str, env: Environment):
 
         # Internal bookkeeping
         self.slice_id: int = 0
-        self.inside_set_or_macro: bool = False
+        self.inside_set_or_macro: bool = False  # {% set %} or {% macro %}
+        self.inside_block = False  # {% block %}
         self.stack: List[int] = []
         self.idx_raw: int = 0
 
@@ -211,7 +221,7 @@ def slice_info_for_literal(self, length, prefix="") -> RawSliceInfo:
         """Returns a RawSliceInfo for a literal.
 
         In the alternate template, literals are replaced with a uniquely
-        numbered, easily-to-parse literal. JinjaTracer uses this output as
+        numbered, easy-to-parse literal. JinjaTracer uses this output as
         a "breadcrumb trail" to deduce the execution path through the template.
 
         This is important even if the original literal (i.e. in the raw SQL
@@ -222,13 +232,16 @@ def slice_info_for_literal(self, length, prefix="") -> RawSliceInfo:
         """
         unique_alternate_id = self.next_slice_id()
         alternate_code = f"\0{prefix}{unique_alternate_id}_{length}"
-        return self.make_raw_slice_info(unique_alternate_id, alternate_code)
+        return self.make_raw_slice_info(
+            unique_alternate_id, alternate_code, inside_block=self.inside_block
+        )
 
-    def update_inside_set_or_macro(
+    def update_inside_set_or_macro_or_block(
         self, block_type: str, trimmed_parts: List[str]
     ) -> None:
         """Based on block tag, update whether we're in a set/macro section."""
         if block_type == "block_start" and trimmed_parts[0] in (
+            "block",
             "macro",
             "set",
         ):
@@ -236,11 +249,12 @@ def update_inside_set_or_macro(
             # - {% set variable = value %}
             # - {% set variable %}value{% endset %}
             # https://jinja.palletsprojects.com/en/2.10.x/templates/#block-assignments
-            # When the second format is used, set the field
-            # 'inside_set_or_macro' to True. This info is used elsewhere,
-            # as other code inside these regions require special handling.
-            # (Generally speaking, JinjaTracer ignores the contents of these
-            # blocks, treating them like opaque templated regions.)
+            # When the second format is used, set one of the fields
+            # 'inside_set_or_macro' or 'inside_block' to True. This info is
+            # used elsewhere, as other code inside these regions require
+            # special handling. (Generally speaking, JinjaAnalyzer ignores
+            # the contents of these blocks, treating them like opaque templated
+            # regions.)
             try:
                 # Entering a set/macro block. Build a source string consisting
                 # of just this one Jinja command and see if it parses. If so,
@@ -255,22 +269,33 @@ def update_inside_set_or_macro(
                     isinstance(e.message, str)
                     and "Unexpected end of template" in e.message
                 ):
-                    # It was opening a block, thus we're inside a set or macro.
-                    self.inside_set_or_macro = True
+                    # It was opening a block, thus we're inside a set, macro, or
+                    # block.
+                    if trimmed_parts[0] == "block":
+                        self.inside_block = True
+                    else:
+                        self.inside_set_or_macro = True
                 else:
                     raise  # pragma: no cover
-        elif block_type == "block_end" and (trimmed_parts[0] in ("endmacro", "endset")):
-            # Exiting a set/macro block.
-            self.inside_set_or_macro = False
+        elif block_type == "block_end":
+            if trimmed_parts[0] in ("endmacro", "endset"):
+                # Exiting a set or macro.
+                self.inside_set_or_macro = False
+            elif trimmed_parts[0] == "endblock":
+                # Exiting a {% block %} block.
+                self.inside_block = False
 
     def make_raw_slice_info(
-        self, unique_alternate_id: Optional[str], alternate_code: Optional[str]
+        self,
+        unique_alternate_id: Optional[str],
+        alternate_code: Optional[str],
+        inside_block: bool = False,
     ) -> RawSliceInfo:
         """Create RawSliceInfo as given, or "empty" if in set/macro block."""
         if not self.inside_set_or_macro:
-            return RawSliceInfo(unique_alternate_id, alternate_code, [])
+            return RawSliceInfo(unique_alternate_id, alternate_code, [], inside_block)
         else:
-            return RawSliceInfo(None, None, [])
+            return RawSliceInfo(None, None, [], False)
 
     # We decide the "kind" of element we're dealing with using its _closing_
     # tag rather than its opening tag. The types here map back to similar types
@@ -351,7 +376,7 @@ def analyze(self, make_template: Callable[[str], Template]) -> JinjaTracer:
                         raw_slice_info = self.track_templated(
                             m_open, m_close, tag_contents
                         )
-                self.update_inside_set_or_macro(block_type, tag_contents)
+                self.update_inside_set_or_macro_or_block(block_type, tag_contents)
                 m_strip_right = regex.search(
                     r"\s+$", raw, regex.MULTILINE | regex.DOTALL
                 )


Test patch:
diff --git a/test/core/templaters/jinja_test.py b/test/core/templaters/jinja_test.py
--- a/test/core/templaters/jinja_test.py
+++ b/test/core/templaters/jinja_test.py
@@ -1060,6 +1060,61 @@ def test__templater_jinja_slice_template(test, result):
                 ("block_end", slice(27, 39, None), slice(13, 13, None)),
             ],
         ),
+        (
+            # Test for issue 3434: Handle {% block %}.
+            "SELECT {% block table_name %}block_contents{% endblock %} "
+            "FROM {{ self.table_name() }}\n",
+            None,
+            [
+                ("literal", slice(0, 7, None), slice(0, 7, None)),
+                ("literal", slice(29, 43, None), slice(7, 21, None)),
+                ("block_start", slice(7, 29, None), slice(21, 21, None)),
+                ("literal", slice(29, 43, None), slice(21, 21, None)),
+                ("block_end", slice(43, 57, None), slice(21, 21, None)),
+                ("literal", slice(57, 63, None), slice(21, 27, None)),
+                ("templated", slice(63, 86, None), slice(27, 27, None)),
+                ("literal", slice(29, 43, None), slice(27, 41, None)),
+                ("literal", slice(86, 87, None), slice(41, 42, None)),
+            ],
+        ),
+        (
+            # Another test for issue 3434: Similar to the first, but uses
+            # the block inside a loop.
+            """{% block table_name %}block_contents{% endblock %}
+SELECT
+{% for j in [4, 5, 6] %}
+FROM {{ j }}{{ self.table_name() }}
+{% endfor %}
+""",
+            None,
+            [
+                ("literal", slice(22, 36, None), slice(0, 14, None)),
+                ("block_start", slice(0, 22, None), slice(14, 14, None)),
+                ("literal", slice(22, 36, None), slice(14, 14, None)),
+                ("block_end", slice(36, 50, None), slice(14, 14, None)),
+                ("literal", slice(50, 58, None), slice(14, 22, None)),
+                ("block_start", slice(58, 82, None), slice(22, 22, None)),
+                ("literal", slice(82, 88, None), slice(22, 28, None)),
+                ("templated", slice(88, 95, None), slice(28, 29, None)),
+                ("templated", slice(95, 118, None), slice(29, 29, None)),
+                ("literal", slice(22, 36, None), slice(29, 43, None)),
+                ("literal", slice(118, 119, None), slice(43, 44, None)),
+                ("block_end", slice(119, 131, None), slice(44, 44, None)),
+                ("literal", slice(82, 88, None), slice(44, 50, None)),
+                ("templated", slice(88, 95, None), slice(50, 51, None)),
+                ("templated", slice(95, 118, None), slice(51, 51, None)),
+                ("literal", slice(22, 36, None), slice(51, 65, None)),
+                ("literal", slice(118, 119, None), slice(65, 66, None)),
+                ("block_end", slice(119, 131, None), slice(66, 66, None)),
+                ("literal", slice(82, 88, None), slice(66, 72, None)),
+                ("templated", slice(88, 95, None), slice(72, 73, None)),
+                ("templated", slice(95, 118, None), slice(73, 73, None)),
+                ("literal", slice(22, 36, None), slice(73, 87, None)),
+                ("literal", slice(118, 119, None), slice(87, 88, None)),
+                ("block_end", slice(119, 131, None), slice(88, 88, None)),
+                ("literal", slice(131, 132, None), slice(88, 89, None)),
+            ],
+        ),
     ],
 )
 def test__templater_jinja_slice_file(raw_file, override_context, result, caplog):