Repository: sqlfluff/sqlfluff
Pull request: More rules logging and L003 indent improvement.
This resolves #734 (or at least part of it).

This fix for the original query is now:

```sql
SELECT
    *
FROM foo
INNER JOIN bar
    USING (baz)
WHERE qux = 1
```

however, if the `bar` is on a new line it still fixes to:

```sql
SELECT
    *
FROM foo
INNER JOIN
        bar
    USING (baz)
WHERE qux = 1
```

I've also tried to add a load more logging in. @barrywhart - does that also make the structure easier to understand?



Problem statement:
sqlfluff fix uses incorrect indentation for `USING` keyword
-- If this is a parsing or linting issue, please include a minimal SQL example which reproduces the issue, along with the `sqlfluff parse` output, `sqlfluff lint` output and `sqlfluff fix` output when relevant. Finally, please add any labels that are related to the issue before submitting so that it can be addressed easily! --
`sqlfluff fix` seems to indent the `USING` keyword three levels deep.

```
SELECT
  *
FROM foo
INNER JOIN bar
  USING (baz)
WHERE qux = 1
```
This code has two areas with 2 space indentation, and those should be fixed accordingly.

## Expected Behaviour
```
SELECT
    *
FROM foo
INNER JOIN bar
    USING (baz)
WHERE qux = 1
```
## Observed Behaviour
```
SELECT
    *
FROM foo
INNER JOIN bar
           USING (baz)
WHERE qux = 1
```

## Steps to Reproduce
`sqlfluff fix`

## Version
Include the output of `sqlfluff --version` along with your Python version
sqlfluff, version 0.4.0a3
Python 3.7.4

## Configuration
```
Include your SQLFluff configuration here
```

Hints from issue discussion:
I ran a related test, using this SQL (same as above, but omitting the word `INNER`):

```
SELECT
  *
FROM foo
JOIN bar
  USING (baz)
WHERE qux = 1
```

The result was:
```
SELECT
    *
FROM foo
JOIN bar
     USING (baz)
WHERE qux = 1
```

I'm not familiar with the details of the indentation algorithm, although the basics are documented [here](https://docs.sqlfluff.com/en/stable/indentation.html). Maybe it's aligning `USING` with the end of the `JOIN` or `INNER JOIN`? @alanmcruickshank, can you please clarify if this is behaving as intended?
Yeah this is a bug. `L003` is generating a fix with five spaces rather than the expected four on the `USING` line. The fix for the `*` in the first line is as expected. This feels like a great case for some test driven development.
I did some investigation of this (my first time looking in detail at L003). If the input file has the `USING` line indented by a multiple of 4 spaces (0, 4, 8), the output is correct. If it's not, L003 applies a "hanging" indent (aligning `USING` with the table name on the previous line) rather than forcing it to the expected indent.

This happens because the `USING` line is not considered a `clean_indent`:
```
                    # If we have a clean indent, we can just add steps in line
                    # with the difference in the indent buffers. simples.
                    # We can also do this if we've skipped a line. I think?
                    if this_line["clean_indent"] or this_line_no - k > 1:
                        desired_indent = "".join(
                            elem.raw for elem in res[k]["indent_buffer"]
                        ) + (self._make_indent() * indent_diff)
                    # If we have the option of a hanging indent then use it.
                    elif res[k]["hanging_indent"]:
                        desired_indent = " " * res[k]["hanging_indent"]
                    else:
                        raise RuntimeError(
                            "Unexpected case, please report bug, including the query you are linting!"
                        )
```

At this point, things became more complex because I wasn't sure what direction to pursue:
* Should line 5 be considered `clean_indent`? Probably not, but I wasn't sure.
* Should this line be changed, and if so, how? `if this_line["clean_indent"] or this_line_no - k > 1:`
* Is it possible that hanging indent should not be allowed here? IIRC, hanging indent is primarily targeted at expressions, which potentially contain many functions, operators, parentheses, etc., where hanging indent helps readability. By comparison, `JOIN` clauses are fairly simple in structure.

Any thoughts on this, @alanmcruickshank?
As an experiment, I split the `JOIN` across lines to see how `sqlfluff fix` indents it.

Before:
```
SELECT
    *
FROM foo
INNER JOIN
  bar
  USING (baz)
WHERE qux = 1
```

After:
```
SELECT
    *
FROM foo
JOIN
        bar
    USING (baz)
WHERE qux = 1
```

This looks wild! I think it's doing a hanging indent, aligning `bar` with `foo` two lines above. (I didn't dig into the code to confirm this.)

Commit message: Linting

Patch to review:
diff --git a/src/sqlfluff/core/rules/base.py b/src/sqlfluff/core/rules/base.py
index 457aaff3e7c..3200e35c5c5 100644
--- a/src/sqlfluff/core/rules/base.py
+++ b/src/sqlfluff/core/rules/base.py
@@ -313,6 +313,8 @@ def crawl(
             )
             return vs, raw_stack, fixes, memory
 
+        new_lerrs = []
+        new_fixes = []
         if res is None:
             # Assume this means no problems (also means no memory)
             pass
@@ -321,8 +323,8 @@ def crawl(
             memory = res.memory
             lerr = res.to_linting_error(rule=self)
             if lerr:
-                vs.append(lerr)
-            fixes += res.fixes
+                new_lerrs = [lerr]
+            new_fixes = res.fixes
         elif isinstance(res, list) and all(
             isinstance(elem, LintResult) for elem in res
         ):
@@ -332,8 +334,8 @@ def crawl(
             for elem in res:
                 lerr = elem.to_linting_error(rule=self)
                 if lerr:
-                    vs.append(lerr)
-                fixes += elem.fixes
+                    new_lerrs.append(lerr)
+                new_fixes += elem.fixes
         else:
             raise TypeError(
                 "Got unexpected result [{0!r}] back from linting rule: {1!r}".format(
@@ -341,6 +343,15 @@ def crawl(
                 )
             )
 
+        for lerr in new_lerrs:
+            self.logger.debug("!! Violation Found: %r", lerr.description)
+        for fix in new_fixes:
+            self.logger.debug("!! Fix Proposed: %r", fix)
+
+        # Consume the new results
+        vs += new_lerrs
+        fixes += new_fixes
+
         # The raw stack only keeps track of the previous raw segments
         if len(segment.segments) == 0:
             raw_stack += (segment,)
diff --git a/src/sqlfluff/core/rules/std/L003.py b/src/sqlfluff/core/rules/std/L003.py
index 060ec8b21c1..47c5153bba8 100644
--- a/src/sqlfluff/core/rules/std/L003.py
+++ b/src/sqlfluff/core/rules/std/L003.py
@@ -43,23 +43,26 @@ class Rule_L003(BaseRule):
     _works_on_unparsable = False
     config_keywords = ["tab_space_size", "indent_unit", "lint_templated_tokens"]
 
-    def _make_indent(self, num=1, tab_space_size=None, indent_unit=None):
-        if (indent_unit or self.indent_unit) == "tab":
+    @staticmethod
+    def _make_indent(num=1, tab_space_size=4, indent_unit="space"):
+        if indent_unit == "tab":
             base_unit = "\t"
-        elif (indent_unit or self.indent_unit) == "space":
-            base_unit = " " * (tab_space_size or self.tab_space_size)
+        elif indent_unit == "space":
+            base_unit = " " * tab_space_size
         return base_unit * num
 
-    def _indent_size(self, segments):
+    @staticmethod
+    def _indent_size(segments, tab_space_size=4):
         indent_size = 0
         for elem in segments:
             raw = elem.raw
             # convert to spaces for convenience (and hanging indents)
-            raw = raw.replace("\t", " " * self.tab_space_size)
+            raw = raw.replace("\t", " " * tab_space_size)
             indent_size += len(raw)
         return indent_size
 
-    def _process_raw_stack(self, raw_stack):
+    @classmethod
+    def _process_raw_stack(cls, raw_stack, tab_space_size=4):
         """Take the raw stack, split into lines and evaluate some stats."""
         indent_balance = 0
         line_no = 1
@@ -123,12 +126,16 @@ def _process_raw_stack(self, raw_stack):
                 else:
                     in_indent = False
                     this_indent_balance = indent_balance
-                    indent_size = self._indent_size(indent_buffer)
+                    indent_size = cls._indent_size(
+                        indent_buffer, tab_space_size=tab_space_size
+                    )
             elif elem.is_meta and elem.indent_val != 0:
                 indent_balance += elem.indent_val
                 if elem.indent_val > 0:
                     # Keep track of the indent at the last ... indent
-                    line_indent_stack.append(self._indent_size(line_buffer))
+                    line_indent_stack.append(
+                        cls._indent_size(line_buffer, tab_space_size=tab_space_size)
+                    )
                     hanger_pos = None
                 else:
                     # this is a dedent, we could still have a hanging indent,
@@ -137,7 +144,9 @@ def _process_raw_stack(self, raw_stack):
                         line_indent_stack.pop()
             elif elem.is_code:
                 if hanger_pos is None:
-                    hanger_pos = self._indent_size(line_buffer[:-1])
+                    hanger_pos = cls._indent_size(
+                        line_buffer[:-1], tab_space_size=tab_space_size
+                    )
 
         # If we get to the end, and still have a buffer, add it on
         if line_buffer:
@@ -146,7 +155,7 @@ def _process_raw_stack(self, raw_stack):
                 "line_buffer": line_buffer,
                 "indent_buffer": indent_buffer,
                 "indent_size": indent_size,
-                "indent_balance": indent_balance,
+                "indent_balance": this_indent_balance,
                 "hanging_indent": line_indent_stack.pop()
                 if line_indent_stack
                 else None,
@@ -185,6 +194,15 @@ def _coerce_indent_to(self, desired_indent, current_indent_buffer, current_ancho
             ]
         return fixes
 
+    @staticmethod
+    def _strip_buffers(line_dict):
+        """Strip a line dict of buffers for logging."""
+        return {
+            key: line_dict[key]
+            for key in line_dict
+            if key not in ("line_buffer", "indent_buffer")
+        }
+
     def _eval(self, segment, raw_stack, memory, **kwargs):
         """Indentation not consistent with previous lines.
 
@@ -242,18 +260,16 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
             # Not in indent and not a newline, don't trigger here.
             return LintResult(memory=memory)
 
-        res = self._process_raw_stack(raw_stack + (segment,))
+        res = self._process_raw_stack(
+            raw_stack + (segment,), tab_space_size=self.tab_space_size
+        )
         this_line_no = max(res.keys())
         this_line = res.pop(this_line_no)
         self.logger.debug(
             "Evaluating line #%s. %s",
             this_line_no,
             # Don't log the line or indent buffer, it's too noisy.
-            {
-                key: this_line[key]
-                for key in this_line
-                if key not in ("line_buffer", "indent_buffer")
-            },
+            self._strip_buffers(this_line),
         )
 
         # Is this line just comments?
@@ -299,6 +315,9 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
                 # This is a HANGER
                 memory["hanging_lines"].append(this_line_no)
                 self.logger.debug("    Hanger Line. #%s", this_line_no)
+                self.logger.debug(
+                    "    Last Line: %s", self._strip_buffers(res[last_code_line])
+                )
                 return LintResult(memory=memory)
 
         # Is this an indented first line?
@@ -355,7 +374,10 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
                     if res[k]["indent_size"] == 0:
                         desired_indent = ""
                     elif this_line["indent_size"] == 0:
-                        desired_indent = self._make_indent()
+                        desired_indent = self._make_indent(
+                            indent_unit=self.indent_unit,
+                            tab_space_size=self.tab_space_size,
+                        )
                     else:
                         # The previous indent.
                         desired_indent = "".join(
@@ -383,31 +405,45 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
             # Are we at a deeper indent?
             elif indent_diff > 0:
                 self.logger.debug("    [deeper indent balance] Comparing to #%s", k)
-                # NB: We shouldn't need to deal with hanging indents
-                # here, they should already have been dealt with before.
+                # NB: We shouldn't need to deal with correct hanging indents
+                # here, they should already have been dealt with before. We
+                # may still need to deal with *creating* hanging indents if
+                # appropriate.
+                self.logger.debug(
+                    "    Comparison Line: %s", self._strip_buffers(res[k])
+                )
 
                 # Check to see if we've got a whole number of multiples. If
                 # we do then record the number for later, otherwise raise
                 # an error. We do the comparison here so we have a reference
                 # point to do the repairs. We need a sensible previous line
-                # to base the repairs off.
+                # to base the repairs off. If there's no indent at all, then
+                # we should also take this route because there SHOULD be one.
                 if this_line["indent_size"] % self.tab_space_size != 0:
                     memory["problem_lines"].append(this_line_no)
 
+                    # The default indent is the one just reconstructs it from
+                    # the indent size.
+                    default_indent = "".join(
+                        elem.raw for elem in res[k]["indent_buffer"]
+                    ) + self._make_indent(
+                        indent_unit=self.indent_unit,
+                        tab_space_size=self.tab_space_size,
+                        num=indent_diff,
+                    )
                     # If we have a clean indent, we can just add steps in line
                     # with the difference in the indent buffers. simples.
                     # We can also do this if we've skipped a line. I think?
                     if this_line["clean_indent"] or this_line_no - k > 1:
-                        desired_indent = "".join(
-                            elem.raw for elem in res[k]["indent_buffer"]
-                        ) + (self._make_indent() * indent_diff)
+                        self.logger.debug("        Use clean indent.")
+                        desired_indent = default_indent
                     # If we have the option of a hanging indent then use it.
                     elif res[k]["hanging_indent"]:
+                        self.logger.debug("        Use hanging indent.")
                         desired_indent = " " * res[k]["hanging_indent"]
                     else:
-                        raise RuntimeError(
-                            "Unexpected case, please report bug, including the query you are linting!"
-                        )
+                        self.logger.debug("        Use default indent.")
+                        desired_indent = default_indent
 
                     # Make fixes
                     fixes = self._coerce_indent_to(
@@ -420,7 +456,7 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
                         anchor=segment,
                         memory=memory,
                         description=(
-                            "Indentation not hanging or " "a multiple of {0} spaces"
+                            "Indentation not hanging or a multiple of {0} spaces"
                         ).format(self.tab_space_size),
                         fixes=fixes,
                     )
@@ -482,7 +518,10 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
                                     "create",
                                     segment,
                                     self.make_whitespace(
-                                        raw=self._make_indent(),
+                                        raw=self._make_indent(
+                                            indent_unit=self.indent_unit,
+                                            tab_space_size=self.tab_space_size,
+                                        ),
                                         pos_marker=segment.pos_marker,
                                     ),
                                 )
@@ -503,7 +542,9 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
                                 self.make_whitespace(
                                     # Make the minimum indent for it to be ok.
                                     raw=self._make_indent(
-                                        num=comp_indent_num - this_indent_num
+                                        num=comp_indent_num - this_indent_num,
+                                        indent_unit=self.indent_unit,
+                                        tab_space_size=self.tab_space_size,
                                     ),
                                     pos_marker=segment.pos_marker,
                                 ),
@@ -513,7 +554,9 @@ def _eval(self, segment, raw_stack, memory, **kwargs):
                 elif this_indent_num > comp_indent_num + indent_diff:
                     # Calculate the lowest ok indent:
                     desired_indent = self._make_indent(
-                        num=comp_indent_num - this_indent_num
+                        num=comp_indent_num - this_indent_num,
+                        indent_unit=self.indent_unit,
+                        tab_space_size=self.tab_space_size,
                     )
 
                     # Make fixes
diff --git a/src/sqlfluff/core/rules/std/L016.py b/src/sqlfluff/core/rules/std/L016.py
index 24c0d902811..f1e7b762464 100644
--- a/src/sqlfluff/core/rules/std/L016.py
+++ b/src/sqlfluff/core/rules/std/L016.py
@@ -80,7 +80,10 @@ def generate_fixes_to_coerce(
                 fixes = []
 
                 # Generate some sample indents:
-                unit_indent = crawler._make_indent()
+                unit_indent = crawler._make_indent(
+                    indent_unit=crawler.indent_unit,
+                    tab_space_size=crawler.tab_space_size,
+                )
                 indent_p1 = indent_section.raw + unit_indent
                 if unit_indent in indent_section.raw:
                     indent_m1 = indent_section.raw.replace(unit_indent, "", 1)
diff --git a/test/conftest.py b/test/conftest.py
index 7beeb65df74..7d18f7276f9 100644
--- a/test/conftest.py
+++ b/test/conftest.py
@@ -4,7 +4,7 @@
 import oyaml
 
 from sqlfluff.core.parser.markers import FilePositionMarker
-from sqlfluff.core.parser.segments import RawSegment
+from sqlfluff.core.parser.segments import RawSegment, Indent, Dedent
 
 
 def process_struct(obj):
@@ -74,6 +74,13 @@ def generate_test_segments_func(elems):
         buff = []
         raw_buff = ""
         for elem in elems:
+            if elem == "<indent>":
+                buff.append(Indent(FilePositionMarker().advance_by(raw_buff)))
+                continue
+            elif elem == "<dedent>":
+                buff.append(Dedent(FilePositionMarker().advance_by(raw_buff)))
+                continue
+
             if set(elem) <= {" ", "\t"}:
                 cls = RawSegment.make(" ", name="whitespace", type="whitespace")
             elif set(elem) <= {"\n"}:
diff --git a/test/core/rules/__init__.py b/test/core/rules/__init__.py
index 0f122212d40..ed9e71502c1 100644
--- a/test/core/rules/__init__.py
+++ b/test/core/rules/__init__.py
@@ -1 +1,7 @@
-"""Tests for the rules module."""
+"""Tests for the rules module.
+
+Where possible, tests should be contained within the
+yaml test cases. For python based tests where this is
+not possible, the rule code should be in the test file
+name.
+"""
diff --git a/test/core/rules/std_L003_test.py b/test/core/rules/std_L003_test.py
new file mode 100644
index 00000000000..be9634a4286
--- /dev/null
+++ b/test/core/rules/std_L003_test.py
@@ -0,0 +1,173 @@
+"""Tests the python routines within L003."""
+
+import pytest
+
+from sqlfluff.core.config import FluffConfig
+from sqlfluff.core.rules.std.L003 import Rule_L003
+from sqlfluff.testing.rules import get_rule_from_set
+
+
+@pytest.mark.parametrize(
+    "test_elems,result",
+    [
+        # Unindented test examples
+        (
+            ["bar"],
+            {
+                1: {
+                    "line_no": 1,
+                    "indent_size": 0,
+                    "indent_balance": 0,
+                    "hanging_indent": None,
+                    "clean_indent": False,
+                },
+            },
+        ),
+        (
+            ["bar", "\n", "     ", "foo", "baar", " \t "],
+            {
+                1: {
+                    "line_no": 1,
+                    "indent_size": 0,
+                    "indent_balance": 0,
+                    "hanging_indent": None,
+                    "clean_indent": False,
+                },
+                2: {
+                    "line_no": 2,
+                    "indent_size": 5,
+                    "indent_balance": 0,
+                    "hanging_indent": None,
+                    "clean_indent": False,
+                },
+            },
+        ),
+        # Indented Test Example
+        (
+            ["bar", "<indent>", "\n", "    ", "foo", "<dedent>", "\n", "baar"],
+            {
+                1: {
+                    "line_no": 1,
+                    "indent_size": 0,
+                    "indent_balance": 0,
+                    "hanging_indent": None,
+                    "clean_indent": False,
+                },
+                2: {
+                    "line_no": 2,
+                    "indent_size": 4,
+                    # Balance of indents is 1 at the start of content
+                    # of this line.
+                    "indent_balance": 1,
+                    "hanging_indent": None,
+                    # It's a clean indent, because there was an indent
+                    # token just before the previous newline.
+                    "clean_indent": True,
+                },
+                3: {
+                    "line_no": 3,
+                    "indent_size": 0,
+                    "indent_balance": 0,
+                    "hanging_indent": None,
+                    "clean_indent": False,
+                },
+            },
+        ),
+        # Hanging Test Example
+        (
+            ["bar", " ", "<indent>", "foo", "\n", "    ", "foo", "<dedent>"],
+            {
+                1: {
+                    "line_no": 1,
+                    "indent_size": 0,
+                    "indent_balance": 0,
+                    # It's a hanging indent possibility because there was an unresolved
+                    # indent in the previous line before the end.
+                    "hanging_indent": 4,
+                    "clean_indent": False,
+                },
+                2: {
+                    "line_no": 2,
+                    "indent_size": 4,
+                    "indent_balance": 1,
+                    "hanging_indent": None,
+                    "clean_indent": False,
+                },
+            },
+        ),
+    ],
+)
+def test__rules__std_L003_process_raw_stack(generate_test_segments, test_elems, result):
+    """Test the _process_raw_stack function.
+
+    Note: This test probably needs expanding. It doesn't
+    really check enough of the full functionality.
+
+    """
+    cfg = FluffConfig()
+    r = get_rule_from_set("L003", config=cfg)
+    test_stack = generate_test_segments(test_elems)
+    res = r._process_raw_stack(test_stack)
+    print(res)
+    # Verify structure
+    assert isinstance(res, dict)
+    assert all(isinstance(k, int) for k in res.keys())
+    assert all(isinstance(v, dict) for v in res.values())
+    # Check keys are all present
+    assert all(
+        v.keys()
+        == {
+            "line_no",
+            "line_buffer",
+            "indent_buffer",
+            "indent_size",
+            "indent_balance",
+            "hanging_indent",
+            "clean_indent",
+        }
+        for v in res.values()
+    )
+    # For testing purposes, we won't be checking the buffer fields. They're just
+    # too hard to create in the test cases and aren't critical in determining
+    # what course of action to take. Most of the logic uses the values which we
+    # *are* still testing.
+    for k in res:
+        del res[k]["line_buffer"]
+        del res[k]["indent_buffer"]
+    assert res == result
+
+
+@pytest.mark.parametrize(
+    "indent_unit,num,tab_space_size,result",
+    [
+        ("space", 3, 2, "      "),
+        ("tab", 3, 2, "\t\t\t"),
+    ],
+)
+def test__rules__std_L003_make_indent(indent_unit, num, tab_space_size, result):
+    """Test Rule_L003._make_indent."""
+    res = Rule_L003._make_indent(
+        num=num, indent_unit=indent_unit, tab_space_size=tab_space_size
+    )
+    assert res == result
+
+
+class ProtoSeg:
+    """Proto Seg for testing."""
+
+    def __init__(self, raw):
+        self.raw = raw
+
+
+@pytest.mark.parametrize(
+    "tab_space_size,segments,result",
+    [
+        # Integer examples
+        (3, [ProtoSeg("      ")], 6),
+        (2, [ProtoSeg("\t\t")], 4),
+    ],
+)
+def test__rules__std_L003_indent_size(tab_space_size, segments, result):
+    """Test Rule_L003._make_indent."""
+    res = Rule_L003._indent_size(segments=segments, tab_space_size=tab_space_size)
+    assert res == result
diff --git a/test/core/rules/std_test.py b/test/core/rules/std_test.py
index 9fe2382d5c8..ccbeb751efb 100644
--- a/test/core/rules/std_test.py
+++ b/test/core/rules/std_test.py
@@ -164,22 +164,6 @@ def test__rules__std_file_dbt(rule, path, violations, in_dbt_project_dir):  # no
     )
 
 
-def test__rules__std_L003_process_raw_stack(generate_test_segments):
-    """Test the _process_raw_stack function.
-
-    Note: This test probably needs expanding. It doesn't
-    really check enough of the full functionality.
-
-    """
-    cfg = FluffConfig()
-    r = get_rule_from_set("L003", config=cfg)
-    test_stack = generate_test_segments(["bar", "\n", "     ", "foo", "baar", " \t "])
-    res = r._process_raw_stack(test_stack)
-    print(res)
-    assert sorted(res.keys()) == [1, 2]
-    assert res[2]["indent_size"] == 5
-
-
 @pytest.mark.parametrize(
     "rule_config_dict",
     [
diff --git a/test/core/rules/test_cases/L003.yml b/test/core/rules/test_cases/L003.yml
index 30fcf47a329..29ea62c73aa 100644
--- a/test/core/rules/test_cases/L003.yml
+++ b/test/core/rules/test_cases/L003.yml
@@ -1,16 +1,17 @@
 rule: L003
 
-test_1:
+test_reindent_first_line_1:
   fail_str: '     SELECT 1'
   fix_str: SELECT 1
 
-test_2:
+test_reindent_first_line_2:
   # Github Bug #99. Python2 Issues with fixing L003
   fail_str: '  select 1 from tbl;'
   fix_str: select 1 from tbl;
 
-test_3:
+test_identation_of_comments_1:
   # Github Bug #203
+  # Comments should be aligned to the following line.
   pass_str: |
     SELECT
         -- Compute the thing
@@ -18,7 +19,8 @@ test_3:
     FROM
         acceptable_buckets
 
-test_4:
+test__identation_of_comments_2:
+  # Comments should be aligned to the following line.
   pass_str: |
     SELECT
         user_id
@@ -33,7 +35,7 @@ test_4:
         USING
             (user_id)
 
-test_5:
+test_tab_indentation:
   # Using tabs as indents works
   fail_str: |
     SELECT
@@ -51,30 +53,26 @@ test_5:
     rules:
       indent_unit: tab
 
-test_6:
+test_indented_joins_default:
   # Configurable indents work.
   # a) default
   pass_str: |
     SELECT a, b, c
-
     FROM my_tbl
-
     LEFT JOIN another_tbl USING(a)
 
-test_7:
+test_indented_joins_false:
   # b) specific
   pass_str: |
     SELECT a, b, c
-
     FROM my_tbl
-
     LEFT JOIN another_tbl USING(a)
 
   configs:
     indentation:
       indented_joins: false
 
-test_8:
+test_indented_joins_true:
   # c) specific True, but passing
   pass_str: |
     SELECT a, b, c
@@ -85,7 +83,7 @@ test_8:
     indentation:
       indented_joins: true
 
-test_9:
+test_indented_joins_true_fix:
   # d) specific True, but failing
   fail_str: |
     SELECT a, b, c
@@ -101,7 +99,7 @@ test_9:
     indentation:
       indented_joins: true
 
-test_10:
+test_indented_joins_false_fix:
   # e) specific False, and failing
   fail_str: |
     SELECT a, b, c
@@ -117,7 +115,7 @@ test_10:
     indentation:
       indented_joins: false
 
-test_11:
+test_indented_from_with_comment:
   pass_str: |
     SELECT *
     FROM
@@ -125,7 +123,7 @@ test_11:
     -- Comment
     JOIN t2 USING (user_id)
 
-test_12:
+test_indented_from_with_comment_fix:
   fail_str: |
     SELECT *
     FROM
@@ -167,3 +165,45 @@ bigquery_with_indent:
   configs:
     core:
       dialect: bigquery
+
+test_attempted_hanger_fix:
+  # It's almost a hanger and so should be corrected to one.
+  fail_str: |
+    SELECT coalesce(foo,
+                  bar)
+       FROM tbl
+
+  fix_str: |
+    SELECT coalesce(foo,
+                    bar)
+    FROM tbl
+
+test_possible_hanger_fix:
+  # It's not even close to a hnaging indent, but we should
+  # still attempt this, although this is potentially a questionable
+  # fix and could be an approach we change in the future.
+  fail_str: |
+    SELECT coalesce(foo,
+     bar)
+       FROM tbl
+
+  fix_str: |
+    SELECT coalesce(foo,
+                    bar)
+    FROM tbl
+
+test_clean_reindent_fix:
+  # A "clean" indent is where the previous line ends with an
+  # indent token (as per this example). We should use the
+  # default approach and indent by 1 step.
+  fail_str: |
+    SELECT coalesce(
+    foo,
+                    bar)
+       FROM tbl
+
+  fix_str: |
+    SELECT coalesce(
+        foo,
+        bar)
+    FROM tbl
diff --git a/test/fixtures/linter/autofix/bigquery/004_templating/after.sql b/test/fixtures/linter/autofix/bigquery/004_templating/after.sql
index 71041dc6727..e4e3038bd9f 100644
--- a/test/fixtures/linter/autofix/bigquery/004_templating/after.sql
+++ b/test/fixtures/linter/autofix/bigquery/004_templating/after.sql
@@ -24,6 +24,8 @@ raw_effect_sizes AS (
     SELECT
         COUNT(1) AS campaign_count_{{action}},
         {{corr_states}}
+    -- NOTE: The L003 fix routine behaves a little strangely here around the templated
+        -- code. This is a bug currently with no obvious solution.
         , SAFE_DIVIDE(SAFE_MULTIPLY(CORR({{metric}}_rate_su, {{action}}), STDDEV_POP({{metric}}_rate_su)),
                         STDDEV_POP({{action}})) AS {{metric}}_{{action}}
     FROM
diff --git a/test/fixtures/linter/autofix/bigquery/004_templating/before.sql b/test/fixtures/linter/autofix/bigquery/004_templating/before.sql
index 3610c163d7d..fcb8942e361 100644
--- a/test/fixtures/linter/autofix/bigquery/004_templating/before.sql
+++ b/test/fixtures/linter/autofix/bigquery/004_templating/before.sql
@@ -24,6 +24,8 @@ WITH
   SELECT
     COUNT(1) AS campaign_count_{{action}},
     {{corr_states}}
+    -- NOTE: The L003 fix routine behaves a little strangely here around the templated
+    -- code. This is a bug currently with no obvious solution.
     ,SAFE_DIVIDE(SAFE_MULTIPLY(CORR({{metric}}_rate_su, {{action}}), STDDEV_POP({{metric}}_rate_su)),
       STDDEV_POP({{action}})) AS {{metric}}_{{action}}
   FROM


Code review comments:
diff_hunk: @@ -1,24 +1,26 @@
 rule: L003
 
-test_1:
+test_reindent_first_line_1:
   fail_str: '     SELECT 1'
   fix_str: SELECT 1
 
-test_2:
+test_reindent_first_line_2:
   # Github Bug #99. Python2 Issues with fixing L003
   fail_str: '  select 1 from tbl;'
   fix_str: select 1 from tbl;
 
-test_3:
+test_identation_of_comments_1:
   # Github Bug #203
+  # Comments should be aligned to the following line.
   pass_str: |
     SELECT
         -- Compute the thing
         (a + b) AS c
     FROM
         acceptable_buckets
 
-test_4:
+test__identation_of_comments_2:
path: test/core/rules/test_cases/L003.yml
text: @user1:
Typos:
* Two underscores after "test"
* identation -> indentation

diff_hunk: @@ -167,3 +165,45 @@ bigquery_with_indent:
   configs:
     core:
       dialect: bigquery
+
+test_attempted_hanger_fix:
+  # It's almost a hanger and so should be corrected to one.
+  fail_str: |
+    SELECT coalesce(foo,
+                  bar)
+       FROM tbl
+
+  fix_str: |
+    SELECT coalesce(foo,
+                    bar)
+    FROM tbl
+
+test_possible_hanger_fix:
+  # It's not even close to a hnaging indent, but we should
path: test/core/rules/test_cases/L003.yml
text: @user1:
Typo: hnaging -> hanging