Repository: sqlfluff/sqlfluff
Pull request: L016: New option "ignore_comment_clause" to ignore column COMMENTs, etc.
<!--Thanks for adding this feature!-->

<!--Please give the Pull Request a meaningful title for the release notes-->

### Brief summary of the change made
Fixes #2554
<!--Please include `fixes #XXXX` to automatically close any corresponding issue when the pull request is merged. Alternatively if not fully closed you can say `makes progress on #XXXX`.-->


### Are there any other side effects of this change that we should be aware of?


### Pull Request checklist
- [x] Please confirm you have completed any of the necessary steps below.

- Included test cases to demonstrate any code changes, which may be one or more of the following:
  - `.yml` rule test cases in `test/fixtures/rules/std_rule_cases`.
  - `.sql`/`.yml` parser test cases in `test/fixtures/dialects` (note YML files can be auto generated with `tox -e generate-fixture-yml`).
  - Full autofix test cases in `test/fixtures/linter/autofix`.
  - Other.
- Added appropriate documentation for the change.
- Created GitHub issues for any relevant followup/future enhancements if appropriate.


Problem statement:
Update L016 to optionally ignore long COMMENT lines as well
<!--Note: This is for general enhancements to the project. Please use the Bug report template instead to raise parsing/linting/syntax issues for existing supported dialects-->
For example:
```
CREATE OR REPLACE TABLE mytable (
    col1 NUMBER COMMENT 'col1 comment',
    col2 BOOLEAN COMMENT 'very long comment line, exceeding max_line_length...',
    col3 NUMBER COMMENT 'col3 comment'
)

```
sqlfluff-fix throws `L016 | Line is too long` error assuming the col2 comment exceeds max_line_length setting.

Setting `ignore_comment_lines` to True also doesn't help.
Updating each of the offending lines with --noqa directive might not be scalable where there are 100s of files involved.

Hints from issue discussion:
This is definitly something I would like to have too, and I can spend some time in trying to make it.

Only question that I have is "how to name this configuration?". I'm thinking about `ignore_too_long_description`. I don't want to use `comment` to not confuse people between "business comment" like the usage above (that can be used for field or table comments) and "sql comment" which will be ignored by the parser when executing the query (`-- ` or `# ` or `/* ... */`).

In the meantime, I'll work on it with a temporary name.
I think I miss some information like the type and the name of possible segments that are considered as literal comments for field or for table. But, my idea was to add a condition right after these lines:

https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/rules/L016.py#L469-L471

Something like

```python
            if ignore_too_long_description and this_line[-1].type == "?<the_expected_segment_type_for_field/table_comment>?":
                return None
```

Also, this needs to be added on this block as we might have a too long column with a trailing inline comment:

https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/rules/L016.py#L537-L540

What do you think about?
Given that we already have a `ignore_comment_lines`, how about `ignore_inline_column_comments`? I'm hoping we're planning on only ignoring L016 on offending lines and still apply other rules on those lines (other words, not completely ignore the offending lines).

I had a look at this and it's pretty tricky! L016 currently looks at the segments after it sees a new line, so doesn't have the full parse tree so can't see a comment literal being part of a `comment_clause`. So it's gonna be tricky...

Alternative might be to automatically add a `--noqa: L016` comment to the end of the line nut that feels a little hacky. And might interfere with other `noqa`'s a user might have added.

@barrywhart 's having a look at L016 for another issue so he might have some better ideas depending on what he discovers during that work...
See PR #2589. I named the new option `ignore_comment_clause` because some dialects implement comment clauses on things besides columns.

Commit message: Merge branch 'main' into bhart-issue_2554_l016_ignore_comment_clause

Patch to review:
diff --git a/src/sqlfluff/core/default_config.cfg b/src/sqlfluff/core/default_config.cfg
index 4ff278c9c0b..f35cf80da71 100644
--- a/src/sqlfluff/core/default_config.cfg
+++ b/src/sqlfluff/core/default_config.cfg
@@ -92,6 +92,7 @@ ignore_words = None
 [sqlfluff:rules:L016]
 # Line length
 ignore_comment_lines = False
+ignore_comment_clause = False
 
 [sqlfluff:rules:L026]
 # References must be in FROM clause
diff --git a/src/sqlfluff/core/rules/config_info.py b/src/sqlfluff/core/rules/config_info.py
index 582421c09f8..186551720e6 100644
--- a/src/sqlfluff/core/rules/config_info.py
+++ b/src/sqlfluff/core/rules/config_info.py
@@ -81,6 +81,13 @@
             " be ignored when linting line lengths?"
         ),
     },
+    "ignore_comment_clause": {
+        "validation": [True, False],
+        "definition": (
+            "Should comment clauses (e.g. column comments) be ignored"
+            " when linting line lengths?"
+        ),
+    },
     "ignore_words": {
         "definition": ("Comma separated list of words to ignore from rule"),
     },
diff --git a/src/sqlfluff/rules/L016.py b/src/sqlfluff/rules/L016.py
index 965c82f2e62..806ce13d84b 100644
--- a/src/sqlfluff/rules/L016.py
+++ b/src/sqlfluff/rules/L016.py
@@ -10,6 +10,7 @@
 )
 
 from sqlfluff.core.rules.base import LintFix, LintResult, RuleContext
+from sqlfluff.core.rules.functional import sp
 from sqlfluff.core.rules.doc_decorators import (
     document_fix_compatible,
     document_configuration,
@@ -29,6 +30,7 @@ class Rule_L016(Rule_L003):
         "tab_space_size",
         "indent_unit",
         "ignore_comment_lines",
+        "ignore_comment_clause",
     ]
 
     def _eval_line_for_breaks(self, segments: List[RawSegment]) -> List[LintFix]:
@@ -418,11 +420,15 @@ def _compute_segment_length(cls, segment: BaseSegment) -> int:
             # case, compute the length of its contents.
             return len(segment.raw)
 
-    @classmethod
-    def _compute_source_length(cls, segments: Sequence[BaseSegment]) -> int:
+    def _compute_source_length(
+        self, segments: Sequence[BaseSegment], memory: dict
+    ) -> int:
         line_len = 0
         seen_slices = set()
         for segment in segments:
+            if self.ignore_comment_clause and segment in memory["comment_clauses"]:
+                continue
+
             slice = (
                 segment.pos_marker.source_slice.start,
                 segment.pos_marker.source_slice.stop,
@@ -445,7 +451,7 @@ def _compute_source_length(cls, segments: Sequence[BaseSegment]) -> int:
             # vast overcount of the source length.
             if slice not in seen_slices:
                 seen_slices.add(slice)
-                line_len += cls._compute_segment_length(segment)
+                line_len += self._compute_segment_length(segment)
         return line_len
 
     def _eval(self, context: RuleContext) -> Optional[LintResult]:
@@ -458,16 +464,30 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
         # Config type hints
         self.max_line_length: int
         self.ignore_comment_lines: bool
+        self.ignore_comment_clause: bool
 
+        if not context.memory:
+            memory: dict = {"comment_clauses": set()}
+        else:
+            memory = context.memory
         if context.segment.name == "newline":
             # iterate to buffer the whole line up to this point
             this_line = self._gen_line_so_far(context.raw_stack)
         else:
+            if self.ignore_comment_clause and context.segment.is_type(
+                "comment_clause", "comment_equals_clause"
+            ):
+                comment_segment = context.functional.segment.children().first(
+                    sp.is_name("quoted_literal")
+                )
+                if comment_segment:
+                    memory["comment_clauses"].add(comment_segment.get())
+
             # Otherwise we're all good
-            return None
+            return LintResult(memory=memory)
 
         # Now we can work out the line length and deal with the content
-        line_len = self._compute_source_length(this_line)
+        line_len = self._compute_source_length(this_line, memory)
         if line_len > self.max_line_length:
             # Problem, we'll be reporting a violation. The
             # question is, can we fix it?
@@ -486,7 +506,7 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
             # They will remain as unfixable.
             if this_line[-1].type == "placeholder":
                 self.logger.info("Unfixable template segment: %s", this_line[-1])
-                return LintResult(anchor=context.segment)
+                return LintResult(anchor=context.segment, memory=memory)
 
             # Does the line end in an inline comment that we can move back?
             if this_line[-1].name == "inline_comment":
@@ -499,9 +519,9 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
                         "Unfixable inline comment, alone on line: %s", this_line[-1]
                     )
                     if self.ignore_comment_lines:
-                        return LintResult()
+                        return LintResult(memory=memory)
                     else:
-                        return LintResult(anchor=context.segment)
+                        return LintResult(anchor=context.segment, memory=memory)
 
                 self.logger.info(
                     "Attempting move of inline comment at end of line: %s",
@@ -523,7 +543,10 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
                     this_line[-1],
                     cast(RawSegment, context.segment),
                 ]
-                if self._compute_source_length(create_elements) > self.max_line_length:
+                if (
+                    self._compute_source_length(create_elements, memory)
+                    > self.max_line_length
+                ):
                     # The inline comment is NOT on a line by itself, but even if
                     # we move it onto a line by itself, it's still too long. In
                     # this case, the rule should do nothing, otherwise it
@@ -535,19 +558,21 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
                         this_line[-1],
                     )
                     if self.ignore_comment_lines:
-                        return LintResult()
+                        return LintResult(memory=memory)
                     else:
-                        return LintResult(anchor=context.segment)
+                        return LintResult(anchor=context.segment, memory=memory)
                 # Create a newline before this one with the existing comment, an
                 # identical indent AND a terminating newline, copied from the current
                 # target segment.
                 create_buffer = [LintFix.create_before(this_line[0], create_elements)]
                 return LintResult(
-                    anchor=context.segment, fixes=delete_buffer + create_buffer
+                    anchor=context.segment,
+                    fixes=delete_buffer + create_buffer,
+                    memory=memory,
                 )
 
             fixes = self._eval_line_for_breaks(this_line)
             if fixes:
-                return LintResult(anchor=context.segment, fixes=fixes)
-            return LintResult(anchor=context.segment)
-        return LintResult()
+                return LintResult(anchor=context.segment, fixes=fixes, memory=memory)
+            return LintResult(anchor=context.segment, memory=memory)
+        return LintResult(memory=memory)
diff --git a/test/fixtures/rules/std_rule_cases/L016.yml b/test/fixtures/rules/std_rule_cases/L016.yml
index b8911da0f03..fc58d9e6186 100644
--- a/test/fixtures/rules/std_rule_cases/L016.yml
+++ b/test/fixtures/rules/std_rule_cases/L016.yml
@@ -238,3 +238,55 @@ test_issue_1666_line_too_long_unfixable_jinja:
   # "tricking" L016 into trying to split the line, then encountering an internal
   # error.
   fail_str: "{{ config (schema='bronze', materialized='view', sort =['id','number'], dist = 'all', tags =['longlonglonglonglong']) }} \n\nselect 1\n"
+
+test_fail_ignore_comment_clause:
+  fail_str: |
+    CREATE OR REPLACE TABLE mytable (
+        col1 NUMBER COMMENT 'col1 comment',
+        col2 BOOLEAN COMMENT 'very long comment line, exceeding max_line_lengthvery long comment line, exceeding max_line_length very long comment line, exceeding max_line_length very long comment line, exceeding max_line_length',
+        col3 NUMBER COMMENT 'col3 comment'
+    )
+
+test_pass_ignore_comment_clause:
+  pass_str: |
+    CREATE OR REPLACE TABLE mytable (
+        col1 NUMBER COMMENT 'col1 comment',
+        col2 BOOLEAN COMMENT 'very long comment line, exceeding max_line_lengthvery long comment line, exceeding max_line_length very long comment line, exceeding max_line_length very long comment line, exceeding max_line_length',
+        col3 NUMBER COMMENT 'col3 comment'
+    )
+  configs:
+    rules:
+      L016:
+        ignore_comment_clause: True
+
+test_pass_ignore_comment_clause_teradata:
+  pass_str: |
+    comment on table sandbox_db.Org_Descendant is 'very long comment line, exceeding max_line_lengthvery long comment line, exceeding max_line_length very long comment line, exceeding max_line_length very long comment line, exceeding max_line_length';
+  configs:
+    core:
+      dialect: teradata
+    rules:
+      L016:
+        ignore_comment_clause: True
+
+test_pass_ignore_comment_clause_exasol:
+  pass_str: |
+    CREATE TABLE IF NOT EXISTS SCHEM.TAB (
+        ID DECIMAL(18, 0) IDENTITY CONSTRAINT PRIMARY KEY DISABLE COMMENT IS 'without constraint name'
+    ) COMMENT IS 'very long comment line, exceeding max_line_lengthvery long comment line, exceeding max_line_length very long comment line, exceeding max_line_length very long comment line, exceeding max_line_length';
+  configs:
+    core:
+      dialect: exasol
+    rules:
+      L016:
+        ignore_comment_clause: True
+
+test_pass_ignore_comment_clause_snowflake:
+  pass_str: |
+    CREATE TABLE foo_table (bar INTEGER) COMMENT = 'very long comment line, exceeding max_line_lengthvery long comment line, exceeding max_line_length very long comment line, exceeding max_line_length very long comment line, exceeding max_line_length'
+  configs:
+    core:
+      dialect: snowflake
+    rules:
+      L016:
+        ignore_comment_clause: True


Code review comments:
diff_hunk: @@ -29,6 +30,7 @@ class Rule_L016(Rule_L003):
         "tab_space_size",
         "indent_unit",
         "ignore_comment_lines",
+        "ignore_comment_clause",
path: src/sqlfluff/rules/L016.py
text: @user1:
```suggestion
        "ignore_comment_clauses",
```

diff_hunk: @@ -458,16 +464,30 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
         # Config type hints
         self.max_line_length: int
         self.ignore_comment_lines: bool
+        self.ignore_comment_clause: bool
 
+        if not context.memory:
+            memory: dict = {"comment_clauses": set()}
+        else:
+            memory = context.memory
         if context.segment.name == "newline":
             # iterate to buffer the whole line up to this point
             this_line = self._gen_line_so_far(context.raw_stack)
         else:
+            if self.ignore_comment_clause and context.segment.is_type(
path: src/sqlfluff/rules/L016.py
text: @user1:
```suggestion
            if self.ignore_comment_clauses and context.segment.is_type(
```

diff_hunk: @@ -81,6 +81,13 @@
             " be ignored when linting line lengths?"
         ),
     },
+    "ignore_comment_clause": {
path: src/sqlfluff/core/rules/config_info.py
text: @user1:
```suggestion
    "ignore_comment_clauses": {
```

diff_hunk: @@ -458,16 +464,30 @@ def _eval(self, context: RuleContext) -> Optional[LintResult]:
         # Config type hints
         self.max_line_length: int
         self.ignore_comment_lines: bool
+        self.ignore_comment_clause: bool
path: src/sqlfluff/rules/L016.py
text: @user1:
```suggestion
        self.ignore_comment_clauses: bool
```

diff_hunk: @@ -418,11 +420,15 @@ def _compute_segment_length(cls, segment: BaseSegment) -> int:
             # case, compute the length of its contents.
             return len(segment.raw)
 
-    @classmethod
-    def _compute_source_length(cls, segments: Sequence[BaseSegment]) -> int:
+    def _compute_source_length(
+        self, segments: Sequence[BaseSegment], memory: dict
+    ) -> int:
         line_len = 0
         seen_slices = set()
         for segment in segments:
+            if self.ignore_comment_clause and segment in memory["comment_clauses"]:
path: src/sqlfluff/rules/L016.py
text: @user1:
```suggestion
            if self.ignore_comment_clauses and segment in memory["comment_clauses"]:
```

diff_hunk: @@ -92,6 +92,7 @@ ignore_words = None
 [sqlfluff:rules:L016]
 # Line length
 ignore_comment_lines = False
+ignore_comment_clause = False
path: src/sqlfluff/core/default_config.cfg
text: @user1:
Make this consistent with config above

```suggestion
ignore_comment_clauses = False
```