Repository: sqlfluff/sqlfluff

Issue:
Fatal templating error with Jinja templater. Tracer produces odd results.
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### What Happened

Issue found while assessing an Airflow project.

The smallest query I can make which triggers the issue is: 
```sql
SELECT
	{% block table_name %}a{% endblock %}.b
FROM d.{{ self.table_name() }}
```

When running this query through `lint` I get an `AssertionError`, or if running on the more friendly error message PR (#3433) I get: `WARNING    Length of templated file mismatch with final slice: 21 != 19.`.

### Expected Behaviour

This query should slice properly and probably eventually give a jinja error that the required variables are undefined.

### Observed Behaviour

I've dug a little into the error and the sliced file being produced is:

```python
[
    TemplatedFileSlice(slice_type='literal', source_slice=slice(0, 8, None), templated_slice=slice(0, 8, None)),
    TemplatedFileSlice(slice_type='block_start', source_slice=slice(8, 30, None), templated_slice=slice(8, 8, None)),
    TemplatedFileSlice(slice_type='literal', source_slice=slice(30, 31, None), templated_slice=slice(8, 9, None)),
    TemplatedFileSlice(slice_type='block_end', source_slice=slice(31, 45, None), templated_slice=slice(9, 9, None)),
    TemplatedFileSlice(slice_type='literal', source_slice=slice(45, 55, None), templated_slice=slice(9, 19, None)),
    TemplatedFileSlice(slice_type='templated', source_slice=slice(55, 78, None), templated_slice=slice(19, 19, None)),
    TemplatedFileSlice(slice_type='literal', source_slice=slice(78, 79, None), templated_slice=slice(19, 19, None))
]
```

The issue is that while the `source_slice` looks correct for the slices, almost all of the `templated_slices` values have zero length, and importantly the last one doesn't end at position 21.

The rendered file is `SELECT\n\ta.b\nFROM d.a\n` (I've included the escape chars) which is indeed 21 chars long.

@barrywhart I might need your help to work out what's going on with the Jinja tracer here.

### How to reproduce

Run provided query, `main` branch. Set to the `jinja` templater.

### Dialect

dialect is set to `snowflake`, but I don't think we're getting far enough for that to make a difference.

### Version

`main` branch commit `cb6357c540d2d968f766f3a7a4fa16f231cb80e4` (and a few branches derived from it)

### Configuration

N/A

### Are you willing to work on and submit a PR to address the issue?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) Report
> Merging [#3436](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) (2c71e8d) into [main](https://codecov.io/gh/sqlfluff/sqlfluff/commit/8e724ef8906eecce7179f4b7c52d4fc0672e4bd9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) (8e724ef) will **decrease** coverage by `0.02%`.
> The diff coverage is `82.35%`.

```diff
@@             Coverage Diff             @@
##              main    #3436      +/-   ##
===========================================
- Coverage   100.00%   99.97%   -0.03%     
===========================================
  Files          170      170              
  Lines        12862    12872      +10     
===========================================
+ Hits         12862    12869       +7     
- Misses           0        3       +3     
```


| [Impacted Files](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff) | Coverage Δ | |
|---|---|---|
| [src/sqlfluff/core/templaters/slicers/tracer.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff#diff-c3JjL3NxbGZsdWZmL2NvcmUvdGVtcGxhdGVycy9zbGljZXJzL3RyYWNlci5weQ==) | `98.70% <82.35%> (-1.30%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff). Last update [8e724ef...2c71e8d](https://codecov.io/gh/sqlfluff/sqlfluff/pull/3436?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=sqlfluff).

barrywhart: @tunetheweb: I updated per your suggestions and also added a more complex test.
barrywhart: I'm going to go ahead and merge. Can always revisit if something comes up later.

Patch:
diff --git a/src/sqlfluff/core/templaters/slicers/tracer.py b/src/sqlfluff/core/templaters/slicers/tracer.py
index c4525030d..6bbbc699e 100644
--- a/src/sqlfluff/core/templaters/slicers/tracer.py
+++ b/src/sqlfluff/core/templaters/slicers/tracer.py
@@ -40,6 +40,7 @@ class RawSliceInfo:
     unique_alternate_id: Optional[str]
     alternate_code: Optional[str]
     next_slice_indices: List[int] = field(default_factory=list)
+    inside_block: bool = field(default=False)  # {% block %}
 
 
 class JinjaTracer:
@@ -101,13 +102,21 @@ class JinjaTracer:
             alt_id, content_info, literal = value
             target_slice_idx = self.find_slice_index(alt_id)
             slice_length = content_info if literal else len(str(content_info))
-            self.move_to_slice(target_slice_idx, slice_length)
+            target_inside_block = self.raw_slice_info[
+                self.raw_sliced[target_slice_idx]
+            ].inside_block
+            if not target_inside_block:
+                # Normal case: Walk through the template.
+                self.move_to_slice(target_slice_idx, slice_length)
+            else:
+                # {% block %} executes code elsewhere in the template but does
+                # not move there. It's a bit like macro invocation.
+                self.record_trace(slice_length, target_slice_idx)
 
         # TRICKY: The 'append_to_templated' parameter is only used by the dbt
         # templater, passing "\n" for this parameter if we need to add one back.
         # (The Jinja templater does not pass this parameter, so
         # 'append_to_templated' gets the default value of "", empty string.)
-        # we receive the default value of "".) The dbt templater will
         # For more detail, see the comments near the call to slice_file() in
         # plugins/sqlfluff-templater-dbt/sqlfluff_templater_dbt/templater.py.
         templated_str = self.make_template(self.raw_str).render() + append_to_templated
@@ -197,7 +206,8 @@ class JinjaAnalyzer:
 
         # Internal bookkeeping
         self.slice_id: int = 0
-        self.inside_set_or_macro: bool = False
+        self.inside_set_or_macro: bool = False  # {% set %} or {% macro %}
+        self.inside_block = False  # {% block %}
         self.stack: List[int] = []
         self.idx_raw: int = 0
 
@@ -211,7 +221,7 @@ class JinjaAnalyzer:
         """Returns a RawSliceInfo for a literal.
 
         In the alternate template, literals are replaced with a uniquely
-        numbered, easily-to-parse literal. JinjaTracer uses this output as
+        numbered, easy-to-parse literal. JinjaTracer uses this output as
         a "breadcrumb trail" to deduce the execution path through the template.
 
         This is important even if the original literal (i.e. in the raw SQL
@@ -222,13 +232,16 @@ class JinjaAnalyzer:
         """
         unique_alternate_id = self.next_slice_id()
         alternate_code = f"\0{prefix}{unique_alternate_id}_{length}"
-        return self.make_raw_slice_info(unique_alternate_id, alternate_code)
+        return self.make_raw_slice_info(
+            unique_alternate_id, alternate_code, inside_block=self.inside_block
+        )
 
-    def update_inside_set_or_macro(
+    def update_inside_set_or_macro_or_block(
         self, block_type: str, trimmed_parts: List[str]
     ) -> None:
         """Based on block tag, update whether we're in a set/macro section."""
         if block_type == "block_start" and trimmed_parts[0] in (
+            "block",
             "macro",
             "set",
         ):
@@ -236,11 +249,12 @@ class JinjaAnalyzer:
             # - {% set variable = value %}
             # - {% set variable %}value{% endset %}
             # https://jinja.palletsprojects.com/en/2.10.x/templates/#block-assignments
-            # When the second format is used, set the field
-            # 'inside_set_or_macro' to True. This info is used elsewhere,
-            # as other code inside these regions require special handling.
-            # (Generally speaking, JinjaTracer ignores the contents of these
-            # blocks, treating them like opaque templated regions.)
+            # When the second format is used, set one of the fields
+            # 'inside_set_or_macro' or 'inside_block' to True. This info is
+            # used elsewhere, as other code inside these regions require
+            # special handling. (Generally speaking, JinjaAnalyzer ignores
+            # the contents of these blocks, treating them like opaque templated
+            # regions.)
             try:
                 # Entering a set/macro block. Build a source string consisting
                 # of just this one Jinja command and see if it parses. If so,
@@ -255,22 +269,33 @@ class JinjaAnalyzer:
                     isinstance(e.message, str)
                     and "Unexpected end of template" in e.message
                 ):
-                    # It was opening a block, thus we're inside a set or macro.
-                    self.inside_set_or_macro = True
+                    # It was opening a block, thus we're inside a set, macro, or
+                    # block.
+                    if trimmed_parts[0] == "block":
+                        self.inside_block = True
+                    else:
+                        self.inside_set_or_macro = True
                 else:
                     raise  # pragma: no cover
-        elif block_type == "block_end" and (trimmed_parts[0] in ("endmacro", "endset")):
-            # Exiting a set/macro block.
-            self.inside_set_or_macro = False
+        elif block_type == "block_end":
+            if trimmed_parts[0] in ("endmacro", "endset"):
+                # Exiting a set or macro.
+                self.inside_set_or_macro = False
+            elif trimmed_parts[0] == "endblock":
+                # Exiting a {% block %} block.
+                self.inside_block = False
 
     def make_raw_slice_info(
-        self, unique_alternate_id: Optional[str], alternate_code: Optional[str]
+        self,
+        unique_alternate_id: Optional[str],
+        alternate_code: Optional[str],
+        inside_block: bool = False,
     ) -> RawSliceInfo:
         """Create RawSliceInfo as given, or "empty" if in set/macro block."""
         if not self.inside_set_or_macro:
-            return RawSliceInfo(unique_alternate_id, alternate_code, [])
+            return RawSliceInfo(unique_alternate_id, alternate_code, [], inside_block)
         else:
-            return RawSliceInfo(None, None, [])
+            return RawSliceInfo(None, None, [], False)
 
     # We decide the "kind" of element we're dealing with using its _closing_
     # tag rather than its opening tag. The types here map back to similar types
@@ -351,7 +376,7 @@ class JinjaAnalyzer:
                         raw_slice_info = self.track_templated(
                             m_open, m_close, tag_contents
                         )
-                self.update_inside_set_or_macro(block_type, tag_contents)
+                self.update_inside_set_or_macro_or_block(block_type, tag_contents)
                 m_strip_right = regex.search(
                     r"\s+$", raw, regex.MULTILINE | regex.DOTALL
                 )


Test patch:
diff --git a/test/core/templaters/jinja_test.py b/test/core/templaters/jinja_test.py
index ca17c36e8..00568e665 100644
--- a/test/core/templaters/jinja_test.py
+++ b/test/core/templaters/jinja_test.py
@@ -1060,6 +1060,61 @@ FROM SOME_TABLE
                 ("block_end", slice(27, 39, None), slice(13, 13, None)),
             ],
         ),
+        (
+            # Test for issue 3434: Handle {% block %}.
+            "SELECT {% block table_name %}block_contents{% endblock %} "
+            "FROM {{ self.table_name() }}\n",
+            None,
+            [
+                ("literal", slice(0, 7, None), slice(0, 7, None)),
+                ("literal", slice(29, 43, None), slice(7, 21, None)),
+                ("block_start", slice(7, 29, None), slice(21, 21, None)),
+                ("literal", slice(29, 43, None), slice(21, 21, None)),
+                ("block_end", slice(43, 57, None), slice(21, 21, None)),
+                ("literal", slice(57, 63, None), slice(21, 27, None)),
+                ("templated", slice(63, 86, None), slice(27, 27, None)),
+                ("literal", slice(29, 43, None), slice(27, 41, None)),
+                ("literal", slice(86, 87, None), slice(41, 42, None)),
+            ],
+        ),
+        (
+            # Another test for issue 3434: Similar to the first, but uses
+            # the block inside a loop.
+            """{% block table_name %}block_contents{% endblock %}
+SELECT
+{% for j in [4, 5, 6] %}
+FROM {{ j }}{{ self.table_name() }}
+{% endfor %}
+""",
+            None,
+            [
+                ("literal", slice(22, 36, None), slice(0, 14, None)),
+                ("block_start", slice(0, 22, None), slice(14, 14, None)),
+                ("literal", slice(22, 36, None), slice(14, 14, None)),
+                ("block_end", slice(36, 50, None), slice(14, 14, None)),
+                ("literal", slice(50, 58, None), slice(14, 22, None)),
+                ("block_start", slice(58, 82, None), slice(22, 22, None)),
+                ("literal", slice(82, 88, None), slice(22, 28, None)),
+                ("templated", slice(88, 95, None), slice(28, 29, None)),
+                ("templated", slice(95, 118, None), slice(29, 29, None)),
+                ("literal", slice(22, 36, None), slice(29, 43, None)),
+                ("literal", slice(118, 119, None), slice(43, 44, None)),
+                ("block_end", slice(119, 131, None), slice(44, 44, None)),
+                ("literal", slice(82, 88, None), slice(44, 50, None)),
+                ("templated", slice(88, 95, None), slice(50, 51, None)),
+                ("templated", slice(95, 118, None), slice(51, 51, None)),
+                ("literal", slice(22, 36, None), slice(51, 65, None)),
+                ("literal", slice(118, 119, None), slice(65, 66, None)),
+                ("block_end", slice(119, 131, None), slice(66, 66, None)),
+                ("literal", slice(82, 88, None), slice(66, 72, None)),
+                ("templated", slice(88, 95, None), slice(72, 73, None)),
+                ("templated", slice(95, 118, None), slice(73, 73, None)),
+                ("literal", slice(22, 36, None), slice(73, 87, None)),
+                ("literal", slice(118, 119, None), slice(87, 88, None)),
+                ("block_end", slice(119, 131, None), slice(88, 88, None)),
+                ("literal", slice(131, 132, None), slice(88, 89, None)),
+            ],
+        ),
     ],
 )
 def test__templater_jinja_slice_file(raw_file, override_context, result, caplog):