Repository: sqlfluff/sqlfluff
Pull request: Disable rules L026 and L028 for BigQuery by default, with option to reenable
### Brief summary of the change made
Fixes #1503 

### Are there any other side effects of this change that we should be aware of?
Rules L026 and L028 will now be disabled by default for BigQuery which technically is a breaking change. However I suspect most BigQuery users will have already disabled these in config. They can manually reenable if desired.

### Pull Request checklist
- [x] Please confirm you have completed any of the necessary steps below.

- Included test cases to demonstrate any code changes, which may be one or more of the following:
  - `.yml` rule test cases in `test/fixtures/rules/std_rule_cases`.
  - `.sql`/`.yml` parser test cases in `test/fixtures/parser` (note YML files can be auto generated with `python test/generate_parse_fixture_yml.py` or by running `tox` locally).
  - Full autofix test cases in `test/fixtures/linter/autofix`.
  - Other.
- Added appropriate documentation for the change.
- Created GitHub issues for any relevant followup/future enhancements if appropriate.


Problem statement:
L026 error while using bigquery nested field
Consider the following sql:

```sql
WITH commit_info AS (
    SELECT
        commit,
        author.name AS author_name
    FROM `bigquery-public-data.github_repos.commits`
    LIMIT 10
)

SELECT * FROM commit_info
```

## Expected Behaviour
Lint should not fail with L026 error

## Observed Behaviour
Lint fails with following error:
```
L:   4 | P:   9 | L026 | Reference 'author.name' refers to table/view not found
                       | in the FROM clause or found in parent subquery.
```

## Dialect
bigquery

## Version
- sqlfluff, version 0.6.6
- Python 3.9.7

Hints from issue discussion:
I use BigQuery a lot and to be honest I turn off L026 in my `.sqlfluff` config as don't think this rule useful for BigQuery.

Perhaps there's a case to be made to exclude BigQuery from this rule by default? We did a similar change for another rule for postgres in #1482 but that would mean no BigQuery user can benefit from this rule even if they want it (i.e. they don't use structs) so not sure that's a good solution.

Honestly I think configuring it off is the best option.

Decided to disable this rule (and also L028 for similar reasons) in #1504 for BigQuery. There is the option to force enable it for those that want it but agree default should be disabled.

Commit message: Add extra docs for rule L033

Patch to review:
diff --git a/src/sqlfluff/core/default_config.cfg b/src/sqlfluff/core/default_config.cfg
index e85fde9da9a..75c19a9b595 100644
--- a/src/sqlfluff/core/default_config.cfg
+++ b/src/sqlfluff/core/default_config.cfg
@@ -66,6 +66,12 @@ extended_capitalisation_policy = consistent
 [sqlfluff:rules:L016]
 ignore_comment_lines = False
 
+[sqlfluff:rules:L026]
+force_enable = False
+
+[sqlfluff:rules:L028]
+force_enable = False
+
 [sqlfluff:rules:L029]  # Keyword identifiers
 unquoted_identifiers_policy = aliases
 
diff --git a/src/sqlfluff/core/rules/config_info.py b/src/sqlfluff/core/rules/config_info.py
index fc7bc65d999..673a142cfd2 100644
--- a/src/sqlfluff/core/rules/config_info.py
+++ b/src/sqlfluff/core/rules/config_info.py
@@ -42,6 +42,12 @@
         "validation": ["consistent", "qualified", "unqualified"],
         "definition": "The expectation for references in single-table select",
     },
+    "force_enable": {
+        "validation": [True, False],
+        "definition": (
+            "Run this rule even for dialects where this rule is disabled by default"
+        ),
+    },
     "unquoted_identifiers_policy": {
         "validation": ["all", "aliases", "column_aliases"],
         "definition": "Types of unquoted identifiers to flag violations for",
diff --git a/src/sqlfluff/rules/L026.py b/src/sqlfluff/rules/L026.py
index ff175fda715..cade3e5b77b 100644
--- a/src/sqlfluff/rules/L026.py
+++ b/src/sqlfluff/rules/L026.py
@@ -2,12 +2,18 @@
 
 from sqlfluff.core.rules.analysis.select import get_aliases_from_select
 from sqlfluff.core.rules.base import LintResult
+from sqlfluff.core.rules.doc_decorators import document_configuration
 from sqlfluff.rules.L025 import Rule_L020
 
 
+@document_configuration
 class Rule_L026(Rule_L020):
     """References cannot reference objects not present in FROM clause.
 
+    NB: This rule is disabled by default for BigQuery due to its use of
+    structs which trigger false positives. It can be enabled with the
+    `force_enable = True` flag.
+
     | **Anti-pattern**
     | In this example, the reference 'vee' has not been declared.
 
@@ -28,6 +34,8 @@ class Rule_L026(Rule_L020):
 
     """
 
+    config_keywords = ["force_enable"]
+
     @staticmethod
     def _is_bad_tbl_ref(table_aliases, parent_select, tbl_ref):
         """Given a table reference, try to find what it's referring to."""
@@ -76,3 +84,14 @@ def _lint_references_and_aliases(
                     )
                 )
         return violation_buff or None
+
+    def _eval(self, segment, parent_stack, dialect, **kwargs):
+        """Override Rule L020 for dialects that use structs.
+
+        Some dialects use structs (e.g. column.field) which look like
+        table references and so incorrectly trigger this rule.
+        """
+        if dialect.name in ["bigquery"] and not self.force_enable:
+            return LintResult()
+
+        return super()._eval(segment, parent_stack, dialect, **kwargs)
diff --git a/src/sqlfluff/rules/L028.py b/src/sqlfluff/rules/L028.py
index ff2cade3287..52f37a8863a 100644
--- a/src/sqlfluff/rules/L028.py
+++ b/src/sqlfluff/rules/L028.py
@@ -9,6 +9,10 @@
 class Rule_L028(Rule_L025):
     """References should be consistent in statements with a single table.
 
+    NB: This rule is disabled by default for BigQuery due to its use of
+    structs which trigger false positives. It can be enabled with the
+    `force_enable = True` flag.
+
     | **Anti-pattern**
     | In this example, only the field `b` is referenced.
 
@@ -38,7 +42,7 @@ class Rule_L028(Rule_L025):
 
     """
 
-    config_keywords = ["single_table_references"]
+    config_keywords = ["single_table_references", "force_enable"]
 
     def _lint_references_and_aliases(
         self,
@@ -92,3 +96,14 @@ def _lint_references_and_aliases(
             seen_ref_types.add(this_ref_type)
 
         return violation_buff or None
+
+    def _eval(self, segment, parent_stack, dialect, **kwargs):
+        """Override Rule L025 for dialects that use structs.
+
+        Some dialects use structs (e.g. column.field) which look like
+        table references and so incorrectly trigger this rule.
+        """
+        if dialect.name in ["bigquery"] and not self.force_enable:
+            return LintResult()
+
+        return super()._eval(segment, parent_stack, dialect, **kwargs)
diff --git a/src/sqlfluff/rules/L033.py b/src/sqlfluff/rules/L033.py
index b6ebaf95822..ebbbd3e5113 100644
--- a/src/sqlfluff/rules/L033.py
+++ b/src/sqlfluff/rules/L033.py
@@ -8,6 +8,9 @@
 class Rule_L033(BaseRule):
     """UNION [DISTINCT|ALL] is preferred over just UNION.
 
+    NB: This rule is only enabled for dialogs that support UNION DISTINCT
+    (`ansi`, `bigquery`, `hive`, and `mysql`).
+
     | **Anti-pattern**
     | In this example, UNION DISTINCT should be preferred over UNION, because
     | explicit is better than implicit.
diff --git a/test/fixtures/rules/std_rule_cases/L026.yml b/test/fixtures/rules/std_rule_cases/L026.yml
index 2d84ccaaaef..ed3a6df83ea 100644
--- a/test/fixtures/rules/std_rule_cases/L026.yml
+++ b/test/fixtures/rules/std_rule_cases/L026.yml
@@ -7,6 +7,9 @@ test_pass_object_referenced_1:
   configs:
     core:
       dialect: bigquery
+    rules:
+      L026:
+        force_enable: True
 
 test_fail_object_not_referenced_1:
   desc: Name foo is not referenced in FROM clause. It would need to be at the end of identifier in ticks or an alias.
@@ -15,6 +18,9 @@ test_fail_object_not_referenced_1:
   configs:
     core:
       dialect: bigquery
+    rules:
+      L026:
+        force_enable: True
 
 test_fail_object_not_referenced_2:
   # References in WHERE clause
@@ -39,4 +45,38 @@ test_pass_object_referenced_4:
     FROM `example_dataset2.example_table2` AS et2
   configs:
     core:
-      dialect: bigquery
\ No newline at end of file
+      dialect: bigquery
+    rules:
+      L026:
+        force_enable: True
+
+test_pass_object_referenced_5:
+  # Test ambiguous column reference caused by use of BigQuery structure fields.
+  # Here, column,field should not trigger the rule as byu default this is off
+  # for bigquery
+  # https://github.com/sqlfluff/sqlfluff/issues/1503
+  pass_str: |
+    SELECT column.field
+    FROM `example_dataset2.example_table2`
+  configs:
+    core:
+      dialect: bigquery
+
+test_pass_object_referenced_6:
+  # Same test as above but default (ANSI) should trigger
+  fail_str: |
+    SELECT column.field
+    FROM table1
+
+test_pass_object_referenced_7:
+  # Same test as above but for BigQuery but force is
+  # enabled so should fail
+  fail_str: |
+    SELECT column.field
+    FROM `example_dataset2.example_table2`
+  configs:
+    core:
+      dialect: bigquery
+    rules:
+      L026:
+        force_enable: True
diff --git a/test/fixtures/rules/std_rule_cases/L028.yml b/test/fixtures/rules/std_rule_cases/L028.yml
index 0d7b28c3896..6d30ec16a0d 100644
--- a/test/fixtures/rules/std_rule_cases/L028.yml
+++ b/test/fixtures/rules/std_rule_cases/L028.yml
@@ -44,3 +44,24 @@ test_value_table_functions_do_not_require_qualification:
   configs:
     core:
       dialect: bigquery
+    rules:
+      L028:
+        force_enable: True
+
+test_object_references_1:
+  fail_str: SELECT a.bar, b FROM my_tbl
+
+test_object_references_2:
+  pass_str: SELECT a.bar, b FROM my_tbl
+  configs:
+    core:
+      dialect: bigquery
+
+test_object_references_3:
+  fail_str: SELECT a.bar, b FROM my_tbl
+  configs:
+    core:
+      dialect: bigquery
+    rules:
+      L028:
+        force_enable: True


Code review comments:
diff_hunk: @@ -39,4 +45,38 @@ test_pass_object_referenced_4:
     FROM `example_dataset2.example_table2` AS et2
   configs:
     core:
-      dialect: bigquery
\ No newline at end of file
+      dialect: bigquery
+    rules:
+      L026:
+        force_enable: True
+
+test_pass_object_referenced_5:
+  # Test ambiguous column reference caused by use of BigQuery structure fields.
+  # Here, column,field should not trigger the rule as byu default this is off
path: test/fixtures/rules/std_rule_cases/L026.yml
text: @user1:
```suggestion
  # Here, column,field should not trigger the rule as by default this rule is disabled.
```