Repository: sqlfluff/sqlfluff
Pull request: Use pluggy to allow plugin architecture for custom rules
# Using pluggy to allow plugin architecture for custom rules (initially)

Resolves #164 

This PR changes sqlfluff to allow custom rules to be plugged through [pluggy](https://pypi.org/project/pluggy/). I think the same system could be used for dialects.

I added an example of a plugin.

Notes: 
* I changed the naming convention for rule classes to be `Rule_<PluginName: optional>_LXXX` so plugin maintainers don't have to worry about rule codes clashing.
* I added a way for users to include a default configuration for the plugin rules.
* I didn't use a different `RuleSet` for the new rules, I just register the plugin rules to the same RuleSet. Is there any advantage in having multiple RuleSets? I see that the current RuleSet is instantiated with the name "standard" which suggests there would be multiple RuleSets?

TODO:

- [x] Find a better example custom rule (any idea?).
- [x] Resolve the [cyclic dependency](https://github.com/sqlfluff/sqlfluff/compare/master...dmateusp:dmateusp/pluggy_for_rules?expand=1#diff-24adf7a8bcdebb4806459eba75db5ded6f952fac149d885a5d440d9e1840d716R343) between config and plugin_manager better (help?)
- [x] Find a good way to expose our test tooling to plugin maintainers (so they can test rules through the YAML system) (help?)
- [x] Add documentation (help?)


Problem statement:
Enhancement: Allow adding organization-specific SQL rules using a plugin layer.
At Mailchimp, we have certain tables that contain sensitive information. In order to access this information safely, we have a few options:
* Instead of accessing the table directly, use a predefined view that filters out sensitive information
* If using the table directly, use a recommended set of filters, e.g. 

```
WHERE
  LOWER(TRIM(email)) NOT IN (
  SELECT
    email
  FROM
    opted_out_emails)
```

Initially, I think we'd simply check each query individually. In that case, I think it'd be great if we were able to implement these checks as classes similar to those in `src/sqlfluff/rules`. I was thinking a plug-in architecture could be good for this.

Farther in the future, I could imagine doing more sophisticated analysis, such as looking across queries (either CTEs within a single query, or separate queries). In this case, I'm not sure how that would work, but perhaps we'd want to just use SQLFluff as a library, not using the command line. There is _potentially_ a tool idea there as well -- having a tool to output the parse tree in a neutral format like JSON, etc.

Hints from issue discussion:
Well *that* is a very interesting idea! I'd not thought that far out.

Outputting the query as parsed JSON wouldn't be too hard - but I'm more curious about importing sqlfluff as a library and then extending it. Have you got any more detailed ideas about what that might look like?
I think the ideal case is to define SQLFluff as a plugin host, similar to the recent work I did on `diff-quality`. Then other packages could register themselves as defining SQLFluff plugins. Then SQLFluff could easily "discover" those packages and import them. Given the way rules are defined with a decorator, e.g.:

```
@std_rule_set.register
class Rule_L001(BaseCrawler):
```

I think simply having SQLFluff import those libraries would be enough to incorporate them into the list of checks. There might be some additional considerations to make sure the class names or other identifying information doesn't "collide" with built-in checks. But overall, I think it wouldn't be that big of a change.

I only thought of this idea recently when someone on our Security team was looking at some of our queries and asked, did you remember to use "safe" table X rather than unsafe table Y. This could be a common type of requirements at other companies, and being able to handle this would add a lot of value to the tool. General code quality is nice, but security and confidentiality are very important. 
@alanmcruickshank: If it would be helpful in clarifying the proposed approach, I'd be happy to spend a few hours creating a proof of concept. The amount of code and work will be small enough that if it looks like a bad idea, we can replan in another direction.

FWIW, the `pluggy` package I used in `diff-quality` was "spun out" of the popular PyTest test framework. I have used it on two projects now, and I think it's some pretty elegant "glue". 😄 
I think implementing a [pluggly](https://pypi.org/project/pluggy/) style plugin layer for sqlfluff would be awesome to add custom rules. We would totally use that at tails too for some dependency checks.
Totally agree.
Plugins are a great feature for this kind of project. Because every organization has an additional set of checks.
Do you have a timeline for when are you going to release it?

@valerii-vorobiov In theory the _beta_ version of organization rules is now possible. Check out #514 which added functionality for users to add their own custom rules, using the `sqlfluff` crawler. 

Do you think something like this meets your needs?
This is not going to be part of the 0.4.0 release which should happen in the next few weeks.

This *is* however going to be one of the next big features I'd like to tackle. If people are happy to share, I'd love to have some examples here of the kinds of things that we'd like to do with plugins so that we have some easy targets to start with.
Possible use cases for our organization:
* On BigQuery, we prefer `USING` over `JOIN` for brevity and readability reasons, in spite of the general recommendation adopted for SQLFluff core, so we'd flag places where it could be used.
* For privacy reasons, queries should avoid using certain of our internal tables, preferring related views that have been "cleaned" of users who have "opted out" from a GDPR/privacy perspective. If they do access these tables, they must `JOIN` with other tables to "manually" apply these filters.
* (This one is still vague) If the result of a query reads from tables known to contain PII (personally identifiable information), it **must** retain the specific fields (e.g. `user_id`) that allow the output data to be tied back to the source. Maintaining this "lineage" / backtracking info is critical in order to allow this data to later be scanned, filtered, and purged of information for users who have said "stop using my data for machine learning projects". This is a legal requirement for ours and many other businesses.
* If a query accesses an "external" table (i.e. from another GCP "project"), that table reference must be "frozen" to a particular date and time using the BigQuery `FOR SYSTEM_TIME AS OF` syntax. This policy assures that query results are reproducible even if the underlying table data changes
  * backfills
  * a job for day X runs on day X+1 due to lag
  * a job fails and is retried

These are the ideas that are "top of mind". If it'd be helpful, I could take a closer look at some of our projects for other possible ideas. 
Another use case: https://github.com/sqlfluff/sqlfluff/pull/514#issuecomment-745636097

Commit message: Test rules__test_helper skip

Patch to review:
diff --git a/docs/source/developingplugins.rst b/docs/source/developingplugins.rst
new file mode 100644
index 00000000000..aadf4579c7c
--- /dev/null
+++ b/docs/source/developingplugins.rst
@@ -0,0 +1,56 @@
+.. _developingpluginsref:
+
+Developing Plugins
+==================
+
+*SQLFluff* is extensible through "plugins". We use the `pluggy library`_
+to make linting Rules pluggable, which enable users to implement rules that
+are just too "organization specific" to be shared, or too platform specific
+to be included in the core library.
+
+.. _`pluggy library`: https://pluggy.readthedocs.io/en/latest/
+
+Creating a plugin
+-----------------
+
+We have an example plugin in `sqlfluff/plugins/sqlfluff-plugin-example`_
+which you can use as a template.
+
+Few things to note about plugins:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Right now, only Rules can be added through plugins. Each plugin can
+implement multiple Rules.
+
+The name of a plugin should start with *"sqlfluff-plugin"* to be valid.
+
+A plugin may need to include a default configuration if its rules
+are configurable: use plugin default configurations **only for that reason**!
+We advise against overwriting core configurations by using a default
+plugin configuration, as there is no mechanism in place to enforce precendence
+between the core library configs and plugin configs,
+and multiple plugins could clash.
+
+A plugin Rule class name should have the structure:
+"Rule_PluginName_L000". The 'L' can be any letter
+and is meant to categorize rules; you could use the
+letter 'S' to denote rules that enforce security checks
+for example.
+
+A plugin Rule code includes the PluginName,
+so a rule "Rule_L000" in core will have code "L000",
+while "Rule_PluginName_L000" will have code "PluginName_L000".
+Codes are used to display errors, they are also used as configuration keys.
+
+We make it easy for plugin developers to test their rules by
+exposing a testing library in *sqlfluff.testing*.
+
+.. _`sqlfluff/plugins/sqlfluff-plugin-example`: https://github.com/sqlfluff/sqlfluff/tree/master/plugins/sqlfluff-plugin-example
+
+Giving feedback
+---------------
+
+Would you like to have other parts of *SQLFluff* be "pluggable"?
+Tell us about it in a `GitHub issue`_ 😄.
+
+.. _`GitHub issue`: https://github.com/sqlfluff/sqlfluff/issues/new?assignees=&labels=enhancement&template=enhancement.md
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 964ddaa538b..df8ebd97c44 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -59,6 +59,7 @@ Contents
    architecture
    cli
    api
+   developingplugins
    inthewild
    jointhecommunity
 
diff --git a/mypy.ini b/mypy.ini
index 6ad5b1b9a30..5b03e6f40f3 100644
--- a/mypy.ini
+++ b/mypy.ini
@@ -26,3 +26,6 @@ ignore_missing_imports = True
 
 [mypy-dbt.*]
 ignore_missing_imports = True
+
+[mypy-pluggy.*]
+ignore_missing_imports = True
diff --git a/plugins/sqlfluff-plugin-example/MANIFEST.in b/plugins/sqlfluff-plugin-example/MANIFEST.in
new file mode 100644
index 00000000000..1eb895549a9
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/MANIFEST.in
@@ -0,0 +1 @@
+include src/example/plugin_default_config.cfg
diff --git a/plugins/sqlfluff-plugin-example/setup.py b/plugins/sqlfluff-plugin-example/setup.py
new file mode 100644
index 00000000000..169ba227a8a
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/setup.py
@@ -0,0 +1,15 @@
+"""Setup file for example plugin."""
+from setuptools import setup
+
+setup(
+    name="sqlfluff-plugin-example",
+    include_package_data=True,
+    package_dir={"": "src"},
+    packages=[
+        "example",
+    ],
+    # TODO: Change me to the first release
+    # that includes the pluggy work.
+    install_requires="sqlfluff>=0.4.0a3",
+    entry_points={"sqlfluff": ["example = example.rules"]},
+)
diff --git a/plugins/sqlfluff-plugin-example/src/example/__init__.py b/plugins/sqlfluff-plugin-example/src/example/__init__.py
new file mode 100644
index 00000000000..c651355cd4c
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/src/example/__init__.py
@@ -0,0 +1 @@
+"""Example sqlfluff plugin."""
diff --git a/plugins/sqlfluff-plugin-example/src/example/plugin_default_config.cfg b/plugins/sqlfluff-plugin-example/src/example/plugin_default_config.cfg
new file mode 100644
index 00000000000..b0902b409f4
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/src/example/plugin_default_config.cfg
@@ -0,0 +1,2 @@
+[sqlfluff:rules:Example_L001]
+forbidden_columns = bar, baaz
diff --git a/plugins/sqlfluff-plugin-example/src/example/rules.py b/plugins/sqlfluff-plugin-example/src/example/rules.py
new file mode 100644
index 00000000000..7a8d6a898bf
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/src/example/rules.py
@@ -0,0 +1,87 @@
+"""An example of a custom rule implemented through the plugin system."""
+
+from sqlfluff.core.plugin import hookimpl
+from sqlfluff.core.rules.base import (
+    BaseCrawler,
+    LintResult,
+)
+from sqlfluff.core.rules.doc_decorators import (
+    document_fix_compatible,
+    document_configuration,
+)
+from typing import List
+import os.path
+from sqlfluff.core.config import ConfigLoader
+
+
+@hookimpl
+def get_rules() -> List[BaseCrawler]:
+    """Get plugin rules."""
+    return [Rule_Example_L001]
+
+
+@hookimpl
+def load_default_config() -> dict:
+    """Loads the default configuration for the plugin."""
+    return ConfigLoader.get_global().load_default_config_file(
+        file_dir=os.path.dirname(__file__),
+        file_name="plugin_default_config.cfg",
+    )
+
+
+@hookimpl
+def get_configs_info() -> dict:
+    """Get rule config validations and descriptions."""
+    return {
+        "forbidden_columns": {"definition": "A list of column to forbid"},
+    }
+
+
+@document_fix_compatible
+@document_configuration
+class Rule_Example_L001(BaseCrawler):
+    """ORDER BY on these columns is forbidden!
+
+    | **Anti-pattern**
+    | Using ORDER BY one some forbidden columns.
+
+    .. code-block::
+
+        SELECT *
+        FROM foo
+        ORDER BY
+            bar,
+            baz
+
+    | **Best practice**
+    | Do not order by these columns.
+
+    .. code-block::
+
+        SELECT *
+        FROM foo
+        ORDER BY bar
+    """
+
+    config_keywords = ["forbidden_columns"]
+
+    def __init__(self, code, description, **kwargs):
+        """Overwrite __init__ to set config."""
+        super().__init__(code, description, **kwargs)
+        self.forbidden_columns = [
+            col.strip() for col in self.forbidden_columns.split(",")
+        ]
+
+    def _eval(self, segment, raw_stack, **kwargs):
+        """We should not use ORDER BY."""
+        if segment.is_type("orderby_clause"):
+            for seg in segment.segments:
+                col_name = seg.raw.lower()
+                if (
+                    seg.is_type("column_reference")
+                    and col_name in self.forbidden_columns
+                ):
+                    return LintResult(
+                        anchor=seg,
+                        description=f"Column `{col_name}` not allowed in ORDER BY.",
+                    )
diff --git a/plugins/sqlfluff-plugin-example/test/rules/rule_test_cases_test.py b/plugins/sqlfluff-plugin-example/test/rules/rule_test_cases_test.py
new file mode 100644
index 00000000000..d6b936367e6
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/test/rules/rule_test_cases_test.py
@@ -0,0 +1,16 @@
+"""Runs the rule test cases."""
+import os
+import pytest
+from sqlfluff.testing.rules import load_test_cases, rules__test_helper
+
+ids, test_cases = load_test_cases(
+    test_cases_path=os.path.join(
+        os.path.abspath(os.path.dirname(__file__)), "test_cases", "*.yml"
+    )
+)
+
+
+@pytest.mark.parametrize("test_case", test_cases, ids=ids)
+def test__rule_test_case(test_case):
+    """Run the tests."""
+    rules__test_helper(test_case)
diff --git a/plugins/sqlfluff-plugin-example/test/rules/test_cases/Rule_Example_L001.yml b/plugins/sqlfluff-plugin-example/test/rules/test_cases/Rule_Example_L001.yml
new file mode 100644
index 00000000000..b4ec9e24e7a
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/test/rules/test_cases/Rule_Example_L001.yml
@@ -0,0 +1,26 @@
+rule: Example_L001
+
+no_forbidden_col_used:
+  pass_str: |
+    select
+      a,
+      sum(b)
+    from tbl
+    group by a
+    order by a
+
+no_order_by_used:
+  pass_str: |
+    select
+      a,
+      b,
+      c
+    from tbl
+
+forbidden_col_used:
+  fail_str: |
+    select
+      bar,
+      baz
+    from tbl
+    order by bar
diff --git a/setup.py b/setup.py
index 2aa79d9b49e..22d6ff21032 100644
--- a/setup.py
+++ b/setup.py
@@ -109,5 +109,6 @@ def read(*names, **kwargs):
             "sqlfluff = sqlfluff.cli.commands:cli",
         ],
         "diff_cover": ["sqlfluff = sqlfluff.diff_quality_plugin"],
+        "sqlfluff": ["sqlfluff = sqlfluff.core.plugin.lib"],
     },
 )
diff --git a/src/sqlfluff/core/config.py b/src/sqlfluff/core/config.py
index 9a4d2e00b9a..f06701b5bed 100644
--- a/src/sqlfluff/core/config.py
+++ b/src/sqlfluff/core/config.py
@@ -7,6 +7,7 @@
 import configparser
 from typing import Dict, List, Tuple, Any, Optional, Union, Iterable
 from pathlib import Path
+from sqlfluff.core.plugin.host import get_plugin_manager
 
 import appdirs
 
@@ -222,11 +223,9 @@ def _incorporate_vals(ctx: dict, vals: List[Tuple[Tuple[str, ...], Any]]) -> dic
             r[n] = v
         return ctx
 
-    def load_default_config_file(self) -> dict:
+    def load_default_config_file(self, file_dir: str, file_name: str) -> dict:
         """Load the default config file."""
-        elems = self._get_config_elems_from_file(
-            os.path.join(os.path.dirname(__file__), "default_config.cfg")
-        )
+        elems = self._get_config_elems_from_file(os.path.join(file_dir, file_name))
         return self._incorporate_vals({}, elems)
 
     def load_config_at_path(self, path: str) -> dict:
@@ -338,7 +337,7 @@ def __init__(
         self, configs: Optional[dict] = None, overrides: Optional[dict] = None
     ):
         self._overrides = overrides  # We only store this for child configs
-        defaults = ConfigLoader.get_global().load_default_config_file()
+        defaults = nested_combine(*get_plugin_manager().hook.load_default_config())
         self._configs = nested_combine(
             defaults, configs or {"core": {}}, {"core": overrides or {}}
         )
diff --git a/src/sqlfluff/core/plugin/__init__.py b/src/sqlfluff/core/plugin/__init__.py
new file mode 100644
index 00000000000..9e85a1a3020
--- /dev/null
+++ b/src/sqlfluff/core/plugin/__init__.py
@@ -0,0 +1,6 @@
+"""Marker to be imported and used in plugins (and for own implementations)."""
+import pluggy
+
+project_name = "sqlfluff"
+plugin_base_name = f"{project_name}-plugin"
+hookimpl = pluggy.HookimplMarker(plugin_base_name)
diff --git a/src/sqlfluff/core/plugin/hookspecs.py b/src/sqlfluff/core/plugin/hookspecs.py
new file mode 100644
index 00000000000..5fb480cfece
--- /dev/null
+++ b/src/sqlfluff/core/plugin/hookspecs.py
@@ -0,0 +1,21 @@
+"""Defines the specification to implement a plugin."""
+
+import pluggy
+from sqlfluff.core.plugin import plugin_base_name
+
+hookspec = pluggy.HookspecMarker(plugin_base_name)
+
+
+class PluginSpec:
+    """Defines the method signatures for plugin implementations."""
+
+    @hookspec
+    def get_rules(self):
+        """Get plugin rules."""
+
+    def load_default_config(self) -> dict:
+        """Loads the default configuration for the plugin."""
+
+    @hookspec
+    def get_configs_info(self) -> dict:
+        """Get rule config validations and descriptions."""
diff --git a/src/sqlfluff/core/plugin/host.py b/src/sqlfluff/core/plugin/host.py
new file mode 100644
index 00000000000..19ac819cb81
--- /dev/null
+++ b/src/sqlfluff/core/plugin/host.py
@@ -0,0 +1,13 @@
+"""Defines the plugin manager getter."""
+import pluggy
+
+from sqlfluff.core.plugin.hookspecs import PluginSpec
+from sqlfluff.core.plugin import plugin_base_name, project_name
+
+
+def get_plugin_manager() -> pluggy.PluginManager:
+    """Initializes the PluginManager."""
+    pm = pluggy.PluginManager(plugin_base_name)
+    pm.add_hookspecs(PluginSpec)
+    pm.load_setuptools_entrypoints(project_name)
+    return pm
diff --git a/src/sqlfluff/core/plugin/lib.py b/src/sqlfluff/core/plugin/lib.py
new file mode 100644
index 00000000000..37974012db8
--- /dev/null
+++ b/src/sqlfluff/core/plugin/lib.py
@@ -0,0 +1,28 @@
+"""Base implementation for the plugin."""
+
+import os.path
+from sqlfluff.core.config import ConfigLoader
+from sqlfluff.core.plugin import hookimpl
+from sqlfluff.core.rules.config_info import STANDARD_CONFIG_INFO_DICT
+from sqlfluff.core.rules.std import get_rules_from_path
+
+
+@hookimpl
+def get_rules():
+    """Get plugin rules."""
+    return get_rules_from_path()
+
+
+@hookimpl
+def load_default_config() -> dict:
+    """Loads the default configuration for the plugin."""
+    return ConfigLoader.get_global().load_default_config_file(
+        file_dir=os.path.join(os.path.dirname(os.path.dirname(__file__))),
+        file_name="default_config.cfg",
+    )
+
+
+@hookimpl
+def get_configs_info() -> dict:
+    """Get rule config validations and descriptions."""
+    return STANDARD_CONFIG_INFO_DICT
diff --git a/src/sqlfluff/core/rules/__init__.py b/src/sqlfluff/core/rules/__init__.py
index e263278789b..989ba76728d 100644
--- a/src/sqlfluff/core/rules/__init__.py
+++ b/src/sqlfluff/core/rules/__init__.py
@@ -2,16 +2,14 @@
 
 from sqlfluff.core.rules.base import RuleSet
 from sqlfluff.core.rules.config_info import STANDARD_CONFIG_INFO_DICT
-from sqlfluff.core.rules.std import (
-    rules as std_rules,
-)  # Import the standard ruleset list
-
+from sqlfluff.core.plugin.host import get_plugin_manager
 
 std_rule_set = RuleSet(name="standard", config_info=STANDARD_CONFIG_INFO_DICT)
 
-# Iterate through the std rules list and register each rule with the std_rule_set
-for rule in std_rules:
-    std_rule_set.register(rule)
+# Iterate through the rules list and register each rule with the std_rule_set
+for plugin_rules in get_plugin_manager().hook.get_rules():
+    for rule in plugin_rules:
+        std_rule_set.register(rule)
 
 
 def get_ruleset(name: str = "standard") -> RuleSet:
diff --git a/src/sqlfluff/core/rules/base.py b/src/sqlfluff/core/rules/base.py
index d9ff5226ae7..4df07b82412 100644
--- a/src/sqlfluff/core/rules/base.py
+++ b/src/sqlfluff/core/rules/base.py
@@ -16,6 +16,7 @@
 
 import copy
 import logging
+import re
 from collections import namedtuple
 
 from sqlfluff.core.parser import RawSegment, KeywordSegment, BaseSegment, SymbolSegment
@@ -463,15 +464,18 @@ def _validate_config_options(self, config, rule=None):
         Config options can also be checked for a specific rule e.g L010.
         """
         rule_config = config.get_section("rules")
-
         for config_name, info_dict in self.config_info.items():
             config_option = (
                 rule_config.get(config_name)
                 if not rule
                 else rule_config.get(rule).get(config_name)
             )
-            valid_options = info_dict["validation"]
-            if config_option not in valid_options and config_option is not None:
+            valid_options = info_dict.get("validation")
+            if (
+                valid_options
+                and config_option not in valid_options
+                and config_option is not None
+            ):
                 raise ValueError(
                     (
                         "Invalid option '{0}' for {1} configuration. Must be one of {2}"
@@ -482,7 +486,21 @@ def _validate_config_options(self, config, rule=None):
                     )
                 )
 
-    def register(self, cls):
+    @property
+    def valid_rule_name_regex(self):
+        """Defines the accepted pattern for rule names.
+
+        The first group captures the plugin name (optional), which
+        must be capitalized.
+        The second group captures the rule code.
+
+        Examples of valid rule names:
+        * Rule_PluginName_L001
+        * Rule_L001
+        """
+        return re.compile(r"Rule_?([A-Z]{1}[a-zA-Z]+)?_([A-Z][0-9]{3})")
+
+    def register(self, cls, plugin=None):
         """Decorate a class with this to add it to the ruleset.
 
         .. code-block:: python
@@ -502,19 +520,24 @@ def eval(self, **kwargs):
         :exc:`ValueError`.
 
         """
-        elems = cls.__name__.split("_")
+        rule_name_match = self.valid_rule_name_regex.match(cls.__name__)
         # Validate the name
-        if len(elems) != 2 or elems[0] != "Rule" or len(elems[1]) != 4:
+        if not rule_name_match:
             raise ValueError(
                 (
-                    "Tried to register rule on set {0!r} with unexpected " "format: {1}"
+                    "Tried to register rule on set {0!r} with unexpected "
+                    "format: {1}, format should be: Rule_PluginName_L123 (for plugins) "
+                    "or Rule_L123 (for core rules)."
                 ).format(self.name, cls.__name__)
             )
 
-        code = elems[1]
+        plugin_name, code = rule_name_match.groups()
         # If the docstring is multiline, then we extract just summary.
         description = cls.__doc__.split("\n")[0]
 
+        if plugin_name:
+            code = f"{plugin_name}_{code}"
+
         # Keep track of the *class* in the register. Don't instantiate yet.
         if code in self._register:
             raise ValueError(
diff --git a/src/sqlfluff/core/rules/config_info.py b/src/sqlfluff/core/rules/config_info.py
index 3a797518e37..f859ba6efda 100644
--- a/src/sqlfluff/core/rules/config_info.py
+++ b/src/sqlfluff/core/rules/config_info.py
@@ -7,6 +7,8 @@
 as document rule configuration.
 """
 
+from sqlfluff.core.plugin.host import get_plugin_manager
+
 STANDARD_CONFIG_INFO_DICT = {
     "tab_space_size": {
         "validation": range(100),
@@ -72,3 +74,12 @@
         ),
     },
 }
+
+
+def get_config_info() -> dict:
+    """Gets the config from core sqlfluff and sqlfluff plugins and merges them."""
+    plugin_manager = get_plugin_manager()
+    configs_info = plugin_manager.hook.get_configs_info()
+    return {
+        k: v for config_info_dict in configs_info for k, v in config_info_dict.items()
+    }
diff --git a/src/sqlfluff/core/rules/doc_decorators.py b/src/sqlfluff/core/rules/doc_decorators.py
index 7c54571f959..07c9e2ccc84 100644
--- a/src/sqlfluff/core/rules/doc_decorators.py
+++ b/src/sqlfluff/core/rules/doc_decorators.py
@@ -1,6 +1,6 @@
 """A collection of decorators to modify rule docstrings for Sphinx."""
 
-from sqlfluff.core.rules.config_info import STANDARD_CONFIG_INFO_DICT
+from sqlfluff.core.rules.config_info import get_config_info
 from sqlfluff.core.rules.base import rules_logger  # noqa
 
 
@@ -28,7 +28,7 @@ def document_configuration(cls, ruleset="std"):
     options in the docs, from a single source of truth.
     """
     if ruleset == "std":
-        config_info = STANDARD_CONFIG_INFO_DICT
+        config_info = get_config_info()
     else:
         raise (
             NotImplementedError(
@@ -47,9 +47,11 @@ def document_configuration(cls, ruleset="std"):
                         keyword, cls.__name__
                     )
                 )
-            config_doc += "\n    |     `{0}`: {1}. Must be one of {2}.".format(
-                keyword, info_dict["definition"], info_dict["validation"]
+            config_doc += "\n    |     `{0}`: {1}.".format(
+                keyword, info_dict["definition"]
             )
+            if "validation" in info_dict:
+                config_doc += " Must be one of {0}.".format(info_dict["validation"])
             config_doc += "\n    |"
     except AttributeError:
         rules_logger.info("No config_keywords defined for {0}".format(cls.__name__))
diff --git a/src/sqlfluff/core/rules/std/__init__.py b/src/sqlfluff/core/rules/std/__init__.py
index d4eb05e1848..f276ac63d97 100644
--- a/src/sqlfluff/core/rules/std/__init__.py
+++ b/src/sqlfluff/core/rules/std/__init__.py
@@ -6,32 +6,40 @@
 from importlib import import_module
 from glob import glob
 
-# All rule files are expected in the format of L*.py
-rules_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "L*.py")
 
-# Create a rules dictionary for importing in sqlfluff/src/sqlfluff/core/rules/__init__.py
-rules = []
+def get_rules_from_path(
+    # All rule files are expected in the format of L*.py
+    rules_path=os.path.join(os.path.abspath(os.path.dirname(__file__)), "L*.py"),
+    base_module="sqlfluff.core.rules.std",
+):
+    """Reads all of the Rule classes from a path into a list."""
+    # Create a rules dictionary for importing in sqlfluff/src/sqlfluff/core/rules/__init__.py
+    rules = []
 
-# Sphinx effectively runs an import * from this module in rules.rst, so initialise
-# __all__ with an empty list before we populate it with the rule names.
-__all__ = []
+    # Sphinx effectively runs an import * from this module in rules.rst, so initialise
+    # __all__ with an empty list before we populate it with the rule names.
+    __all__ = []
 
-for module in sorted(glob(rules_path)):
-    # Manipulate the module path to extract the filename without the .py
-    rule_id = os.path.splitext(os.path.basename(module))[0]
-    # All rule classes are expected in the format of Rule_L*
-    rule_class_name = f"Rule_{rule_id}"
-    try:
-        rule_class = getattr(
-            import_module(f"sqlfluff.core.rules.std.{rule_id}"), rule_class_name
-        )
-    except AttributeError:
-        raise (AttributeError("Rule classes must be named in the format of L*."))
-    # Add the rules to the rules dictionary for sqlfluff/src/sqlfluff/core/rules/__init__.py
-    rules.append(rule_class)
-    # Add the rule_classes to the module namespace with globals() so that they can
-    # be found by Sphinx automodule documentation in rules.rst
-    # The result is the same as declaring the classes in this file.
-    globals()[rule_class_name] = rule_class
-    # Add the rule class names to __all__ for Sphinx automodule discovery
-    __all__.append(rule_class_name)
+    for module in sorted(glob(rules_path)):
+        # Manipulate the module path to extract the filename without the .py
+        rule_id = os.path.splitext(os.path.basename(module))[0]
+        # All rule classes are expected in the format of Rule_L*
+        rule_class_name = f"Rule_{rule_id}"
+        try:
+            rule_class = getattr(
+                import_module(f"{base_module}.{rule_id}"), rule_class_name
+            )
+        except AttributeError as e:
+            raise AttributeError(
+                "Rule classes must be named in the format of L*."
+            ) from e
+        # Add the rules to the rules dictionary for sqlfluff/src/sqlfluff/core/rules/__init__.py
+        rules.append(rule_class)
+        # Add the rule_classes to the module namespace with globals() so that they can
+        # be found by Sphinx automodule documentation in rules.rst
+        # The result is the same as declaring the classes in this file.
+        globals()[rule_class_name] = rule_class
+        # Add the rule class names to __all__ for Sphinx automodule discovery
+        __all__.append(rule_class_name)
+
+    return rules
diff --git a/src/sqlfluff/testing/__init__.py b/src/sqlfluff/testing/__init__.py
new file mode 100644
index 00000000000..c418a4a8a3d
--- /dev/null
+++ b/src/sqlfluff/testing/__init__.py
@@ -0,0 +1 @@
+"""Testing utils we want to expose for usage by plugins."""
diff --git a/src/sqlfluff/testing/rules.py b/src/sqlfluff/testing/rules.py
new file mode 100644
index 00000000000..77d0d5285c8
--- /dev/null
+++ b/src/sqlfluff/testing/rules.py
@@ -0,0 +1,122 @@
+"""Testing utils for rule plugins."""
+from sqlfluff.core import Linter
+from sqlfluff.core.errors import SQLParseError
+from sqlfluff.core.rules import get_ruleset
+from sqlfluff.core.config import FluffConfig
+from typing import Tuple, List, NamedTuple, Optional
+from glob import glob
+
+import pytest
+import oyaml as yaml
+
+
+class RuleTestCase(NamedTuple):
+    """Used like a dataclass by rule tests."""
+
+    rule: Optional[str] = None
+    desc: Optional[str] = None
+    pass_str: Optional[str] = None
+    fail_str: Optional[str] = None
+    fix_str: Optional[str] = None
+    configs: Optional[dict] = None
+    skip: Optional[str] = None
+
+
+def load_test_cases(
+    test_cases_path: str,
+) -> Tuple[List[str], List[RuleTestCase]]:
+    """Load rule test cases from YAML files."""
+    ids = []
+    test_cases = []
+
+    for path in sorted(glob(test_cases_path)):
+        with open(path) as f:
+            raw = f.read()
+
+        y = yaml.safe_load(raw)
+
+        rule = y.pop("rule")
+        ids.extend([rule + "_" + t for t in y])
+        test_cases.extend([RuleTestCase(rule=rule, **v) for k, v in y.items()])
+
+    return ids, test_cases
+
+
+def get_rule_from_set(code, config):
+    """Fetch a rule from the rule set."""
+    for r in get_ruleset().get_rulelist(config=config):
+        if r.code == code:
+            return r
+    raise ValueError("{0!r} not in {1!r}".format(code, get_ruleset()))
+
+
+def assert_rule_fail_in_sql(code, sql, configs=None):
+    """Assert that a given rule does fail on the given sql."""
+    # Set up the config to only use the rule we are testing.
+    cfg = FluffConfig(configs=configs, overrides={"rules": code})
+    # Lint it using the current config (while in fix mode)
+    linted = Linter(config=cfg).lint_string(sql, fix=True)
+    lerrs = linted.get_violations()
+    print("Errors Found: {0}".format(lerrs))
+    parse_errors = list(filter(lambda v: type(v) == SQLParseError, lerrs))
+    if parse_errors:
+        pytest.fail(f"Found the following parse errors in test case: {parse_errors}")
+    if not any(v.rule.code == code for v in lerrs):
+        pytest.fail(
+            "No {0} failures found in query which should fail.".format(code),
+            pytrace=False,
+        )
+    # The query should already have been fixed if possible so just return the raw.
+    return linted.tree.raw
+
+
+def assert_rule_pass_in_sql(code, sql, configs=None):
+    """Assert that a given rule doesn't fail on the given sql."""
+    # Configs allows overrides if we want to use them.
+    cfg = FluffConfig(configs=configs)
+    r = get_rule_from_set(code, config=cfg)
+    parsed = Linter(config=cfg).parse_string(sql)
+    if parsed.violations:
+        pytest.fail(parsed.violations[0].desc() + "\n" + parsed.tree.stringify())
+    print("Parsed:\n {0}".format(parsed.tree.stringify()))
+    lerrs, _, _, _ = r.crawl(parsed.tree, dialect=cfg.get("dialect_obj"))
+    print("Errors Found: {0}".format(lerrs))
+    if any(v.rule.code == code for v in lerrs):
+        pytest.fail(
+            "Found {0} failures in query which should pass.".format(code), pytrace=False
+        )
+
+
+def assert_rule_raises_violations_in_file(rule, fpath, violations, fluff_config):
+    """Assert that a given rule raises given errors in specific positions of a file."""
+    lntr = Linter(config=fluff_config)
+    lnt = lntr.lint_path(fpath)
+    # Reformat the test data to match the format we're expecting. We use
+    # sets because we really don't care about order and if one is missing,
+    # we don't care about the orders of the correct ones.
+    assert set(lnt.check_tuples()) == {(rule, v[0], v[1]) for v in violations}
+
+
+def rules__test_helper(test_case):
+    """Test that a rule passes/fails on a set of test_cases.
+
+    Optionally, also test the fixed string if provided in the test case.
+    """
+    if test_case.skip:
+        pytest.skip(test_case.skip)
+
+    if test_case.pass_str:
+        assert_rule_pass_in_sql(
+            test_case.rule,
+            test_case.pass_str,
+            configs=test_case.configs,
+        )
+    if test_case.fail_str:
+        res = assert_rule_fail_in_sql(
+            test_case.rule,
+            test_case.fail_str,
+            configs=test_case.configs,
+        )
+        # If a `fixed` value is provided then check it matches
+        if test_case.fix_str:
+            assert res == test_case.fix_str
diff --git a/test/core/plugin_test.py b/test/core/plugin_test.py
new file mode 100644
index 00000000000..58457aab2ca
--- /dev/null
+++ b/test/core/plugin_test.py
@@ -0,0 +1,31 @@
+"""Plugin related tests."""
+from sqlfluff.core.plugin.host import get_plugin_manager
+from sqlfluff.core.config import FluffConfig
+
+
+def test__plugin_manager_registers_example_plugin():
+    """Test that the example plugin is registered."""
+    plugin_manager = get_plugin_manager()
+    # The plugin import order is non-deterministic
+    assert sorted(
+        [plugin_module.__name__ for plugin_module in plugin_manager.get_plugins()]
+    ) == [
+        "example.rules",
+        "sqlfluff.core.plugin.lib",
+    ]
+
+
+def test__plugin_example_rules_returned():
+    """Test that the example rules from the plugin are returned."""
+    plugin_manager = get_plugin_manager()
+    # The plugin import order is non-deterministic
+    assert "Rule_Example_L001" in [
+        rule.__name__ for rules in plugin_manager.hook.get_rules() for rule in rules
+    ]
+
+
+def test__plugin_default_config_read():
+    """Test that the example plugin default config is merged into FluffConfig."""
+    fluff_config = FluffConfig()
+    # The plugin import order is non-deterministic
+    assert "forbidden_columns" in fluff_config._configs["rules"]["Example_L001"]
diff --git a/test/core/rules/rule_test_cases_test.py b/test/core/rules/rule_test_cases_test.py
index 514ed9aedff..2813ac3b662 100644
--- a/test/core/rules/rule_test_cases_test.py
+++ b/test/core/rules/rule_test_cases_test.py
@@ -1,31 +1,20 @@
 """Runs the rule test cases."""
 import os
-from glob import glob
-
 import pytest
-import oyaml as yaml
-
-from sqlfluff.core.config import FluffConfig
+from sqlfluff.testing.rules import (
+    load_test_cases,
+    rules__test_helper,
+    get_rule_from_set,
+)
 from sqlfluff.core.rules.doc_decorators import is_fix_compatible
-from .std_test import get_rule_from_set, rules__test_helper, RuleTestCase
-
-ids = []
-test_cases = []
+from sqlfluff.core.config import FluffConfig
 
-test_cases_path = os.path.join(
-    os.path.abspath(os.path.dirname(__file__)), "test_cases", "*.yml"
+ids, test_cases = load_test_cases(
+    test_cases_path=os.path.join(
+        os.path.abspath(os.path.dirname(__file__)), "test_cases", "*.yml"
+    )
 )
 
-for path in sorted(glob(test_cases_path)):
-    with open(path) as f:
-        raw = f.read()
-
-    y = yaml.safe_load(raw)
-
-    rule = y.pop("rule")
-    ids.extend([rule + "_" + t for t in y])
-    test_cases.extend([RuleTestCase(rule=rule, **v) for k, v in y.items()])
-
 
 @pytest.mark.parametrize("test_case", test_cases, ids=ids)
 def test__rule_test_case(test_case):
diff --git a/test/core/rules/std_test.py b/test/core/rules/std_test.py
index 45529e88048..e35626927c9 100644
--- a/test/core/rules/std_test.py
+++ b/test/core/rules/std_test.py
@@ -1,113 +1,24 @@
 """Tests for the standard set of rules."""
-from typing import NamedTuple
-
 import pytest
 
 from sqlfluff.core import Linter
-from sqlfluff.core.errors import SQLParseError
 from sqlfluff.core.rules.base import BaseCrawler, LintResult, LintFix
-from sqlfluff.core.rules import std_rule_set
+from sqlfluff.core.rules import get_ruleset
 from sqlfluff.core.rules.doc_decorators import document_configuration
 from sqlfluff.core.config import FluffConfig
+from sqlfluff.testing.rules import (
+    assert_rule_raises_violations_in_file,
+    get_rule_from_set,
+)
 
 from test.fixtures.dbt.templater import (  # noqa
     DBT_FLUFF_CONFIG,
     in_dbt_project_dir,
     dbt_templater,
 )
-
-
-class RuleTestCase(NamedTuple):
-    """Used like a dataclass by rule tests."""
-
-    rule: str = None
-    desc: str = None
-    pass_str: str = None
-    fail_str: str = None
-    fix_str: str = None
-    configs: dict = None
-    skip: str = None
-
-
-def get_rule_from_set(code, config):
-    """Fetch a rule from the rule set."""
-    for r in std_rule_set.get_rulelist(config=config):
-        if r.code == code:
-            return r
-    raise ValueError("{0!r} not in {1!r}".format(code, std_rule_set))
-
-
-def assert_rule_fail_in_sql(code, sql, configs=None):
-    """Assert that a given rule does fail on the given sql."""
-    # Set up the config to only use the rule we are testing.
-    cfg = FluffConfig(configs=configs, overrides={"rules": code})
-    # Lint it using the current config (while in fix mode)
-    linted = Linter(config=cfg).lint_string(sql, fix=True)
-    lerrs = linted.get_violations()
-    print("Errors Found: {0}".format(lerrs))
-    parse_errors = list(filter(lambda v: type(v) == SQLParseError, lerrs))
-    if parse_errors:
-        pytest.fail(f"Found the following parse errors in test case: {parse_errors}")
-    if not any(v.rule.code == code for v in lerrs):
-        pytest.fail(
-            "No {0} failures found in query which should fail.".format(code),
-            pytrace=False,
-        )
-    # The query should already have been fixed if possible so just return the raw.
-    return linted.tree.raw
-
-
-def assert_rule_pass_in_sql(code, sql, configs=None):
-    """Assert that a given rule doesn't fail on the given sql."""
-    # Configs allows overrides if we want to use them.
-    cfg = FluffConfig(configs=configs)
-    r = get_rule_from_set(code, config=cfg)
-    parsed = Linter(config=cfg).parse_string(sql)
-    if parsed.violations:
-        pytest.fail(parsed.violations[0].desc() + "\n" + parsed.tree.stringify())
-    print("Parsed:\n {0}".format(parsed.tree.stringify()))
-    lerrs, _, _, _ = r.crawl(parsed.tree, dialect=cfg.get("dialect_obj"))
-    print("Errors Found: {0}".format(lerrs))
-    if any(v.rule.code == code for v in lerrs):
-        pytest.fail(
-            "Found {0} failures in query which should pass.".format(code), pytrace=False
-        )
-
-
-def assert_rule_raises_violations_in_file(rule, fpath, violations, fluff_config):
-    """Assert that a given rule raises given errors in specific positions of a file."""
-    lntr = Linter(config=fluff_config)
-    lnt = lntr.lint_path(fpath)
-    # Reformat the test data to match the format we're expecting. We use
-    # sets because we really don't care about order and if one is missing,
-    # we don't care about the orders of the correct ones.
-    assert set(lnt.check_tuples()) == {(rule, v[0], v[1]) for v in violations}
-
-
-def rules__test_helper(test_case):
-    """Test that a rule passes/fails on a set of test_cases.
-
-    Optionally, also test the fixed string if provided in the test case.
-    """
-    if test_case.skip:
-        pytest.skip(test_case.skip)
-
-    if test_case.pass_str:
-        assert_rule_pass_in_sql(
-            test_case.rule,
-            test_case.pass_str,
-            configs=test_case.configs,
-        )
-    if test_case.fail_str:
-        res = assert_rule_fail_in_sql(
-            test_case.rule,
-            test_case.fail_str,
-            configs=test_case.configs,
-        )
-        # If a `fixed` value is provided then check it matches
-        if test_case.fix_str:
-            assert res == test_case.fix_str
-        return res
+from test.fixtures.rules.L000 import Rule_L000
+from test.fixtures.rules.S000 import Rule_S000
+from sqlfluff.core.rules.std import get_rules_from_path
 
 
 class Rule_T042(BaseCrawler):
@@ -288,7 +199,7 @@ def test_improper_configs_are_rejected(rule_config_dict):
     """Ensure that unsupported configs raise a ValueError."""
     config = FluffConfig(configs={"rules": rule_config_dict})
     with pytest.raises(ValueError):
-        std_rule_set.get_rulelist(config)
+        get_ruleset().get_rulelist(config)
 
 
 def test_rules_cannot_be_instantiated_without_declared_configs():
@@ -328,14 +239,40 @@ class RuleWithoutConfig(BaseCrawler):
 
 def test_rule_exception_is_caught_to_validation():
     """Assert that a rule that throws an exception on _eval returns it as a validation."""
+    std_rule_set = get_ruleset()
 
     @std_rule_set.register
-    class Rule_LXXX(BaseCrawler):
+    class Rule_T000(BaseCrawler):
         """Rule that throws an exception."""
 
         def _eval(self, segment, parent_stack, **kwargs):
             raise Exception("Catch me or I'll deny any linting results from you")
 
-    linter = Linter(config=FluffConfig(overrides=dict(rules="LXXX")))
+    linter = Linter(
+        config=FluffConfig(overrides=dict(rules="T000")),
+        user_rules=[Rule_T000],
+    )
+
+    assert linter.lint_string("select 1").check_tuples() == [("T000", 1, 1)]
+
+
+def test_std_rule_import_fail_bad_naming():
+    """Check that rule import from file works."""
+    assert get_rules_from_path(
+        rules_path="test/fixtures/rules/*.py", base_module="test.fixtures.rules"
+    ) == [Rule_L000, Rule_S000]
+
+    with pytest.raises(AttributeError):
+        get_rules_from_path(
+            rules_path="test/fixtures/rules/bad_rule_name/*.py",
+            base_module="test.fixtures.rules.bad_rule_name",
+        )
+
+
+def test_rule_set_return_informative_error_when_rule_not_registered():
+    """Assert that a rule that throws an exception on _eval returns it as a validation."""
+    cfg = FluffConfig()
+    with pytest.raises(ValueError) as e:
+        get_rule_from_set("L000", config=cfg)
 
-    assert linter.lint_string("select 1").check_tuples() == [("LXXX", 1, 1)]
+    e.match("'L000' not in")
diff --git a/test/fixtures/rules/L000.py b/test/fixtures/rules/L000.py
new file mode 100644
index 00000000000..3dc699d24d5
--- /dev/null
+++ b/test/fixtures/rules/L000.py
@@ -0,0 +1,7 @@
+"""Test std rule import."""
+
+
+class Rule_L000:
+    """Test std rule import."""
+
+    pass
diff --git a/test/fixtures/rules/S000.py b/test/fixtures/rules/S000.py
new file mode 100644
index 00000000000..cc1e81e5c55
--- /dev/null
+++ b/test/fixtures/rules/S000.py
@@ -0,0 +1,7 @@
+"""Test std rule import."""
+
+
+class Rule_S000:
+    """Test std rule import."""
+
+    pass
diff --git a/test/fixtures/rules/bad_rule_name/E000.py b/test/fixtures/rules/bad_rule_name/E000.py
new file mode 100644
index 00000000000..243b2356444
--- /dev/null
+++ b/test/fixtures/rules/bad_rule_name/E000.py
@@ -0,0 +1,7 @@
+"""Test std rule import."""
+
+
+class E000:
+    """This will fail to import because it does not start with Rule_."""
+
+    pass
diff --git a/test/test_testing.py b/test/test_testing.py
new file mode 100644
index 00000000000..8e35232d30a
--- /dev/null
+++ b/test/test_testing.py
@@ -0,0 +1,46 @@
+"""Test the sqlfluff.testing module."""
+
+from _pytest.outcomes import Failed, Skipped
+import pytest
+from sqlfluff.testing.rules import (
+    assert_rule_fail_in_sql,
+    assert_rule_pass_in_sql,
+    rules__test_helper,
+    RuleTestCase,
+)
+
+
+def test_assert_rule_fail_in_sql_handle_parse_error():
+    """Util assert_rule_fail_in_sql should handle parse errors."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_fail_in_sql(code="L000", sql="select from")
+    failed_test.match("Found the following parse errors in test case:")
+
+
+def test_assert_rule_fail_in_sql_should_fail_queries_that_unexpectedly_pass():
+    """Util assert_rule_fail_in_sql should fail tests when a query passes rules that it violates."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_fail_in_sql(code="L001", sql="select 1")
+    failed_test.match("No L001 failures found in query which should fail")
+
+
+def test_assert_rule_pass_in_sql_should_handle_parse_error():
+    """Util assert_rule_pass_in_sql should handle parse errors."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_pass_in_sql(code="L001", sql="select from")
+    failed_test.match("Found unparsable section:")
+
+
+def test_assert_rule_pass_in_sql_should_fail_when_there_are_violations():
+    """Util assert_rule_pass_in_sql should fail when there are violations."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_pass_in_sql(code="L005", sql="select a , b from t")
+    failed_test.match("Found L005 failures in query which should pass")
+
+
+def test_rules__test_helper_skipped_when_test_case_skipped():
+    """Util rules__test_helper should skip the test when test case is "skipped"."""
+    rule_test_case = RuleTestCase(skip="Skip this one for now")
+    with pytest.raises(Skipped) as skipped_test:
+        rules__test_helper(rule_test_case)
+    skipped_test.match("Skip this one for now")
diff --git a/tox.ini b/tox.ini
index 2cd766612b4..3865301199f 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,6 +12,8 @@ deps =
     -Urrequirements_dev.txt
     # Requests is required for the util script
     requests
+    # Add the example plugin
+    plugins/sqlfluff-plugin-example
     # Define dbt versions
     dbt017: dbt==0.17.2
     dbt018: dbt==0.18.1


Code review comments:
diff_hunk: @@ -63,15 +63,17 @@ class Rule_Example_L001(BaseCrawler):
         ORDER BY bar
     """
 
-    # Binary operators behave like keywords too.
-    _target_elems: List[Tuple[str, str]] = [("type", "orderby_clause")]
     config_keywords = ["forbidden_columns"]
 
+    def __init__(self, code, description, **kwargs):
path: plugins/sqlfluff-plugin-example/src/example/rules.py
text: @user1:
Nitpick: Since the class-specific `__init__()` function isn't using `code` or `description`, you could do this instead:
```
def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
```

No big deal, just a bit less code, and avoids having to change this code if the base class parameter list changes.

@author:
Ah makes sense! I changed that and the other nitpick

diff_hunk: @@ -251,3 +254,25 @@ def _eval(self, segment, parent_stack, **kwargs):
     )
 
     assert linter.lint_string("select 1").check_tuples() == [("T000", 1, 1)]
+
+
+def test_std_rule_import_fail_bad_naming():
+    """Check that rule import from file works."""
+    assert get_rules_from_path(
+        rules_path="test/fixtures/rules/*.py", base_module="test.fixtures.rules"
+    ) == [Rule_L000, Rule_S000]
+
+    with pytest.raises(AttributeError):
path: test/core/rules/std_test.py
text: @user1:
Nitpick: Worth checking the error message?