Repository: sqlfluff/sqlfluff

Issue:
2.0.2 - LT02 issues when query contains "do" statement.
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### What Happened

SQLFluff v2.0.2 gives LT02 indentation errors for the Jinja `if`-block when `template_blocks_indent` is set to `True`.
The example SQL below is a bit contrived, but it's the smallest failing example I could produce based on our real SQL.

If I remove the Jinja `do`-expression from the code, the `if` block validates without errors.

### Expected Behaviour

I expect the SQL to pass the linting tests.

### Observed Behaviour

Output from SQLFluff v2.0.2:
```
L:   5 | P:   1 | LT02 | Line should not be indented.
                       | [layout.indent]
L:   6 | P:   1 | LT02 | Line should not be indented.
                       | [layout.indent]
```

### How to reproduce

SQL to reproduce:
```
{% set cols = ['a', 'b'] %}
{% do cols.remove('a') %}

{% if true %}
    select a
    from some_table
{% endif %}
```

### Dialect

`ansi`

### Version

```
> sqlfluff --version
sqlfluff, version 2.0.2

> python --version
Python 3.9.9
```

### Configuration

```
[sqlfluff]
dialect = ansi
templater = jinja

[sqlfluff:indentation]
template_blocks_indent = True
```

### Are you willing to work on and submit a PR to address the issue?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)

2.0.2 - LT02 issues when query contains "do" statement.
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### What Happened

SQLFluff v2.0.2 gives LT02 indentation errors for the Jinja `if`-block when `template_blocks_indent` is set to `True`.
The example SQL below is a bit contrived, but it's the smallest failing example I could produce based on our real SQL.

If I remove the Jinja `do`-expression from the code, the `if` block validates without errors.

### Expected Behaviour

I expect the SQL to pass the linting tests.

### Observed Behaviour

Output from SQLFluff v2.0.2:
```
L:   5 | P:   1 | LT02 | Line should not be indented.
                       | [layout.indent]
L:   6 | P:   1 | LT02 | Line should not be indented.
                       | [layout.indent]
```

### How to reproduce

SQL to reproduce:
```
{% set cols = ['a', 'b'] %}
{% do cols.remove('a') %}

{% if true %}
    select a
    from some_table
{% endif %}
```

### Dialect

`ansi`

### Version

```
> sqlfluff --version
sqlfluff, version 2.0.2

> python --version
Python 3.9.9
```

### Configuration

```
[sqlfluff]
dialect = ansi
templater = jinja

[sqlfluff:indentation]
template_blocks_indent = True
```

### Are you willing to work on and submit a PR to address the issue?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)



Discussion:
I think think this is almost certainly about the `do` statement, hopefully this should be very solvable.
Any pointers on where I should start looking if I would work on a fix @alanmcruickshank?
@fredriv - great question. I just had a quick look and this is a very strange bug, but hopefully one with a satisfying solution.

If I run `sqlfluff parse` on the file I get this:

```
[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% set cols = ['a', 'b'] %}"]
[L:  1, P: 28]      |    newline:                                                  '\n'
[L:  2, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: "{% do cols.remove('a') %}", Block: '230a18']
[L:  2, P: 26]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    newline:                                                  '\n'
[L:  4, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: '{% if true %}', Block: 'e33036']
[L:  4, P: 14]      |    newline:                                                  '\n'
[L:  5, P:  1]      |    whitespace:                                               '    '
[L:  5, P:  5]      |    statement:
[L:  5, P:  5]      |        select_statement:
[L:  5, P:  5]      |            select_clause:
[L:  5, P:  5]      |                keyword:                                      'select'
[L:  5, P: 11]      |                [META] indent:
[L:  5, P: 11]      |                whitespace:                                   ' '
[L:  5, P: 12]      |                select_clause_element:
[L:  5, P: 12]      |                    column_reference:
[L:  5, P: 12]      |                        naked_identifier:                     'a'
[L:  5, P: 13]      |            newline:                                          '\n'
[L:  6, P:  1]      |            whitespace:                                       '    '
[L:  6, P:  5]      |            [META] dedent:
[L:  6, P:  5]      |            from_clause:
[L:  6, P:  5]      |                keyword:                                      'from'
[L:  6, P:  9]      |                whitespace:                                   ' '
[L:  6, P: 10]      |                from_expression:
[L:  6, P: 10]      |                    [META] indent:
[L:  6, P: 10]      |                    from_expression_element:
[L:  6, P: 10]      |                        table_expression:
[L:  6, P: 10]      |                            table_reference:
[L:  6, P: 10]      |                                naked_identifier:             'some_table'
[L:  6, P: 20]      |                    [META] dedent:
[L:  6, P: 20]      |    newline:                                                  '\n'
[L:  7, P:  1]      |    [META] placeholder:                                       [Type: 'block_end', Raw: '{% endif %}', Block: 'e33036']
[L:  7, P: 12]      |    [META] end_of_file:
```

Note the difference between that and the output when I remove the `do` line:

```
[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% set cols = ['a', 'b'] %}"]
[L:  1, P: 28]      |    newline:                                                  '\n'
[L:  2, P:  1]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    newline:                                                  '\n'
[L:  4, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: '{% if true %}', Block: '0d1e98']
[L:  4, P: 14]      |    [META] indent:                                            [Block: '0d1e98']
[L:  4, P: 14]      |    newline:                                                  '\n'
[L:  5, P:  1]      |    whitespace:                                               '    '
[L:  5, P:  5]      |    statement:
[L:  5, P:  5]      |        select_statement:
[L:  5, P:  5]      |            select_clause:
[L:  5, P:  5]      |                keyword:                                      'select'
[L:  5, P: 11]      |                [META] indent:
[L:  5, P: 11]      |                whitespace:                                   ' '
[L:  5, P: 12]      |                select_clause_element:
[L:  5, P: 12]      |                    column_reference:
[L:  5, P: 12]      |                        naked_identifier:                     'a'
[L:  5, P: 13]      |            newline:                                          '\n'
[L:  6, P:  1]      |            whitespace:                                       '    '
[L:  6, P:  5]      |            [META] dedent:
[L:  6, P:  5]      |            from_clause:
[L:  6, P:  5]      |                keyword:                                      'from'
[L:  6, P:  9]      |                whitespace:                                   ' '
[L:  6, P: 10]      |                from_expression:
[L:  6, P: 10]      |                    [META] indent:
[L:  6, P: 10]      |                    from_expression_element:
[L:  6, P: 10]      |                        table_expression:
[L:  6, P: 10]      |                            table_reference:
[L:  6, P: 10]      |                                naked_identifier:             'some_table'
[L:  6, P: 20]      |                    [META] dedent:
[L:  6, P: 20]      |    newline:                                                  '\n'
[L:  7, P:  1]      |    [META] dedent:                                            [Block: '0d1e98']
[L:  7, P:  1]      |    [META] placeholder:                                       [Type: 'block_end', Raw: '{% endif %}', Block: '0d1e98']
[L:  7, P: 12]      |    [META] end_of_file:
```

See that in the latter example there are `indent` and `dedent` tokens around the `if` clause, but not in the first example. Something about the `do` call is disrupting the positioning of those indent tokens. Those tokens are inserted during `._iter_segments()` in `lexer.py`, and more specifically in `._handle_zero_length_slice()`. That's probably where you'll find the issue. My guess is that something about the `do` block is throwing off the block tracking?
Thanks! I'll see if I can have a look at it tonight.

Could it have something to do with the `do` block not having a corresponding `block_end`? 🤔
So perhaps it should be `templated` instead of `block_start`, similar to the `set` above it?
If I add `do` to the list of tag names in `extract_block_type` at https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/core/templaters/slicers/tracer.py#L527 it regards it as a `templated` element instead of `block_start`, and the indent is added where I want it.

E.g.
```
[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% set cols = ['a', 'b'] %}"]
[L:  1, P: 28]      |    newline:                                                  '\n'
[L:  2, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% do cols.remove('a') %}"]
[L:  2, P: 26]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    newline:                                                  '\n'
[L:  4, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: '{% if true %}', Block: '3e39bd']
[L:  4, P: 14]      |    [META] indent:                                            [Block: '3e39bd']
[L:  4, P: 14]      |    newline:                                                  '\n'
[L:  5, P:  1]      |    whitespace:                                               '    '
[L:  5, P:  5]      |    statement:
[L:  5, P:  5]      |        select_statement:
[L:  5, P:  5]      |            select_clause:
[L:  5, P:  5]      |                keyword:                                      'select'
[L:  5, P: 11]      |                [META] indent:
[L:  5, P: 11]      |                whitespace:                                   ' '
[L:  5, P: 12]      |                select_clause_element:
[L:  5, P: 12]      |                    column_reference:
[L:  5, P: 12]      |                        naked_identifier:                     'a'
[L:  5, P: 13]      |            newline:                                          '\n'
[L:  6, P:  1]      |            whitespace:                                       '    '
[L:  6, P:  5]      |            [META] dedent:
[L:  6, P:  5]      |            from_clause:
[L:  6, P:  5]      |                keyword:                                      'from'
[L:  6, P:  9]      |                whitespace:                                   ' '
[L:  6, P: 10]      |                from_expression:
[L:  6, P: 10]      |                    [META] indent:
[L:  6, P: 10]      |                    from_expression_element:
[L:  6, P: 10]      |                        table_expression:
[L:  6, P: 10]      |                            table_reference:
[L:  6, P: 10]      |                                naked_identifier:             'some_table'
[L:  6, P: 20]      |                    [META] dedent:
[L:  6, P: 20]      |    newline:                                                  '\n'
[L:  7, P:  1]      |    [META] dedent:                                            [Block: '3e39bd']
[L:  7, P:  1]      |    [META] placeholder:                                       [Type: 'block_end', Raw: '{% endif %}', Block: '3e39bd']
[L:  7, P: 12]      |    newline:                                                  '\n'
[L:  8, P:  1]      |    [META] end_of_file:
```
Similarly if I instead add `do` to the list of trimmed parts in `update_inside_set_call_macro_or_block` at https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/core/templaters/slicers/tracer.py#L252-L255

Maybe a better place to put it? What do you think @alanmcruickshank?
@fredriv - based on the [docs for jinja](https://jinja.palletsprojects.com/en/3.0.x/extensions/#expression-statement) it looks like we should never get a "do block" (i.e. `{% do ... %} ... {% enddo %}`, it's only ever just `{% do ... %}`). That means that treating it like a `templated` section is the right route, i.e. we should add it in `extract_block_type` and not in `update_inside_set_call_macro_or_block`.

Thanks for your research - I think this should be a neat solution! 🚀 
👍 Ok, I can make a PR :)
I think think this is almost certainly about the `do` statement, hopefully this should be very solvable.
Any pointers on where I should start looking if I would work on a fix @alanmcruickshank?
@fredriv - great question. I just had a quick look and this is a very strange bug, but hopefully one with a satisfying solution.

If I run `sqlfluff parse` on the file I get this:

```
[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% set cols = ['a', 'b'] %}"]
[L:  1, P: 28]      |    newline:                                                  '\n'
[L:  2, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: "{% do cols.remove('a') %}", Block: '230a18']
[L:  2, P: 26]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    newline:                                                  '\n'
[L:  4, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: '{% if true %}', Block: 'e33036']
[L:  4, P: 14]      |    newline:                                                  '\n'
[L:  5, P:  1]      |    whitespace:                                               '    '
[L:  5, P:  5]      |    statement:
[L:  5, P:  5]      |        select_statement:
[L:  5, P:  5]      |            select_clause:
[L:  5, P:  5]      |                keyword:                                      'select'
[L:  5, P: 11]      |                [META] indent:
[L:  5, P: 11]      |                whitespace:                                   ' '
[L:  5, P: 12]      |                select_clause_element:
[L:  5, P: 12]      |                    column_reference:
[L:  5, P: 12]      |                        naked_identifier:                     'a'
[L:  5, P: 13]      |            newline:                                          '\n'
[L:  6, P:  1]      |            whitespace:                                       '    '
[L:  6, P:  5]      |            [META] dedent:
[L:  6, P:  5]      |            from_clause:
[L:  6, P:  5]      |                keyword:                                      'from'
[L:  6, P:  9]      |                whitespace:                                   ' '
[L:  6, P: 10]      |                from_expression:
[L:  6, P: 10]      |                    [META] indent:
[L:  6, P: 10]      |                    from_expression_element:
[L:  6, P: 10]      |                        table_expression:
[L:  6, P: 10]      |                            table_reference:
[L:  6, P: 10]      |                                naked_identifier:             'some_table'
[L:  6, P: 20]      |                    [META] dedent:
[L:  6, P: 20]      |    newline:                                                  '\n'
[L:  7, P:  1]      |    [META] placeholder:                                       [Type: 'block_end', Raw: '{% endif %}', Block: 'e33036']
[L:  7, P: 12]      |    [META] end_of_file:
```

Note the difference between that and the output when I remove the `do` line:

```
[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% set cols = ['a', 'b'] %}"]
[L:  1, P: 28]      |    newline:                                                  '\n'
[L:  2, P:  1]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    newline:                                                  '\n'
[L:  4, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: '{% if true %}', Block: '0d1e98']
[L:  4, P: 14]      |    [META] indent:                                            [Block: '0d1e98']
[L:  4, P: 14]      |    newline:                                                  '\n'
[L:  5, P:  1]      |    whitespace:                                               '    '
[L:  5, P:  5]      |    statement:
[L:  5, P:  5]      |        select_statement:
[L:  5, P:  5]      |            select_clause:
[L:  5, P:  5]      |                keyword:                                      'select'
[L:  5, P: 11]      |                [META] indent:
[L:  5, P: 11]      |                whitespace:                                   ' '
[L:  5, P: 12]      |                select_clause_element:
[L:  5, P: 12]      |                    column_reference:
[L:  5, P: 12]      |                        naked_identifier:                     'a'
[L:  5, P: 13]      |            newline:                                          '\n'
[L:  6, P:  1]      |            whitespace:                                       '    '
[L:  6, P:  5]      |            [META] dedent:
[L:  6, P:  5]      |            from_clause:
[L:  6, P:  5]      |                keyword:                                      'from'
[L:  6, P:  9]      |                whitespace:                                   ' '
[L:  6, P: 10]      |                from_expression:
[L:  6, P: 10]      |                    [META] indent:
[L:  6, P: 10]      |                    from_expression_element:
[L:  6, P: 10]      |                        table_expression:
[L:  6, P: 10]      |                            table_reference:
[L:  6, P: 10]      |                                naked_identifier:             'some_table'
[L:  6, P: 20]      |                    [META] dedent:
[L:  6, P: 20]      |    newline:                                                  '\n'
[L:  7, P:  1]      |    [META] dedent:                                            [Block: '0d1e98']
[L:  7, P:  1]      |    [META] placeholder:                                       [Type: 'block_end', Raw: '{% endif %}', Block: '0d1e98']
[L:  7, P: 12]      |    [META] end_of_file:
```

See that in the latter example there are `indent` and `dedent` tokens around the `if` clause, but not in the first example. Something about the `do` call is disrupting the positioning of those indent tokens. Those tokens are inserted during `._iter_segments()` in `lexer.py`, and more specifically in `._handle_zero_length_slice()`. That's probably where you'll find the issue. My guess is that something about the `do` block is throwing off the block tracking?
Thanks! I'll see if I can have a look at it tonight.

Could it have something to do with the `do` block not having a corresponding `block_end`? 🤔
So perhaps it should be `templated` instead of `block_start`, similar to the `set` above it?
If I add `do` to the list of tag names in `extract_block_type` at https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/core/templaters/slicers/tracer.py#L527 it regards it as a `templated` element instead of `block_start`, and the indent is added where I want it.

E.g.
```
[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% set cols = ['a', 'b'] %}"]
[L:  1, P: 28]      |    newline:                                                  '\n'
[L:  2, P:  1]      |    [META] placeholder:                                       [Type: 'templated', Raw: "{% do cols.remove('a') %}"]
[L:  2, P: 26]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    newline:                                                  '\n'
[L:  4, P:  1]      |    [META] placeholder:                                       [Type: 'block_start', Raw: '{% if true %}', Block: '3e39bd']
[L:  4, P: 14]      |    [META] indent:                                            [Block: '3e39bd']
[L:  4, P: 14]      |    newline:                                                  '\n'
[L:  5, P:  1]      |    whitespace:                                               '    '
[L:  5, P:  5]      |    statement:
[L:  5, P:  5]      |        select_statement:
[L:  5, P:  5]      |            select_clause:
[L:  5, P:  5]      |                keyword:                                      'select'
[L:  5, P: 11]      |                [META] indent:
[L:  5, P: 11]      |                whitespace:                                   ' '
[L:  5, P: 12]      |                select_clause_element:
[L:  5, P: 12]      |                    column_reference:
[L:  5, P: 12]      |                        naked_identifier:                     'a'
[L:  5, P: 13]      |            newline:                                          '\n'
[L:  6, P:  1]      |            whitespace:                                       '    '
[L:  6, P:  5]      |            [META] dedent:
[L:  6, P:  5]      |            from_clause:
[L:  6, P:  5]      |                keyword:                                      'from'
[L:  6, P:  9]      |                whitespace:                                   ' '
[L:  6, P: 10]      |                from_expression:
[L:  6, P: 10]      |                    [META] indent:
[L:  6, P: 10]      |                    from_expression_element:
[L:  6, P: 10]      |                        table_expression:
[L:  6, P: 10]      |                            table_reference:
[L:  6, P: 10]      |                                naked_identifier:             'some_table'
[L:  6, P: 20]      |                    [META] dedent:
[L:  6, P: 20]      |    newline:                                                  '\n'
[L:  7, P:  1]      |    [META] dedent:                                            [Block: '3e39bd']
[L:  7, P:  1]      |    [META] placeholder:                                       [Type: 'block_end', Raw: '{% endif %}', Block: '3e39bd']
[L:  7, P: 12]      |    newline:                                                  '\n'
[L:  8, P:  1]      |    [META] end_of_file:
```
Similarly if I instead add `do` to the list of trimmed parts in `update_inside_set_call_macro_or_block` at https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/core/templaters/slicers/tracer.py#L252-L255

Maybe a better place to put it? What do you think @alanmcruickshank?
@fredriv - based on the [docs for jinja](https://jinja.palletsprojects.com/en/3.0.x/extensions/#expression-statement) it looks like we should never get a "do block" (i.e. `{% do ... %} ... {% enddo %}`, it's only ever just `{% do ... %}`). That means that treating it like a `templated` section is the right route, i.e. we should add it in `extract_block_type` and not in `update_inside_set_call_macro_or_block`.

Thanks for your research - I think this should be a neat solution! 🚀 
👍 Ok, I can make a PR :)

Patch:
diff --git a/src/sqlfluff/core/linter/linter.py b/src/sqlfluff/core/linter/linter.py
--- a/src/sqlfluff/core/linter/linter.py
+++ b/src/sqlfluff/core/linter/linter.py
@@ -189,7 +189,7 @@ def _lex_templated_file(
                 getattr(elem, "indent_val", 0)
                 for elem in cast(Tuple[BaseSegment, ...], tokens)
             )
-            if indent_balance != 0:
+            if indent_balance != 0:  # pragma: no cover
                 linter_logger.debug(
                     "Indent balance test failed for %r. Template indents will not be "
                     "linted for this file.",
@@ -207,7 +207,7 @@ def _lex_templated_file(
                 if token.indent_val != 0:
                     # Don't allow it if we're not linting templating block indents.
                     if not templating_blocks_indent:
-                        continue
+                        continue  # pragma: no cover
             new_tokens.append(token)
 
         # Return new buffer
diff --git a/src/sqlfluff/core/templaters/slicers/tracer.py b/src/sqlfluff/core/templaters/slicers/tracer.py
--- a/src/sqlfluff/core/templaters/slicers/tracer.py
+++ b/src/sqlfluff/core/templaters/slicers/tracer.py
@@ -524,7 +524,7 @@ def extract_block_type(tag_name, block_subtype):
         # a block, but its behavior is basically syntactic sugar for
         # {{ open("somefile).read() }}. Thus, treat it as templated code.
         # It's a similar situation with {% import %} and {% from ... import %}.
-        if tag_name in ["include", "import", "from"]:
+        if tag_name in ["include", "import", "from", "do"]:
             block_type = "templated"
         elif tag_name.startswith("end"):
             block_type = "block_end"


Test patch:
diff --git a/test/core/templaters/jinja_test.py b/test/core/templaters/jinja_test.py
--- a/test/core/templaters/jinja_test.py
+++ b/test/core/templaters/jinja_test.py
@@ -1091,6 +1091,23 @@ def _load_result(*args, **kwargs):
                 ("literal", slice(132, 133, None), slice(34, 35, None)),
             ],
         ),
+        (
+            # Tests Jinja "do" directive. Should be treated as a
+            # templated instead of block - issue 4603.
+            """{% do true %}
+
+{% if true %}
+    select 1
+{% endif %}""",
+            None,
+            [
+                ("templated", slice(0, 13, None), slice(0, 0, None)),
+                ("literal", slice(13, 15, None), slice(0, 2, None)),
+                ("block_start", slice(15, 28, None), slice(2, 2, None)),
+                ("literal", slice(28, 42, None), slice(2, 16, None)),
+                ("block_end", slice(42, 53, None), slice(16, 16, None)),
+            ],
+        ),
         (
             # Tests issue 2541, a bug where the {%- endfor %} was causing
             # IndexError: list index out of range.