Repository: sqlfluff/sqlfluff

Issue:
Enhancement: Allow adding organization-specific SQL rules using a plugin layer.
At Mailchimp, we have certain tables that contain sensitive information. In order to access this information safely, we have a few options:
* Instead of accessing the table directly, use a predefined view that filters out sensitive information
* If using the table directly, use a recommended set of filters, e.g. 

```
WHERE
  LOWER(TRIM(email)) NOT IN (
  SELECT
    email
  FROM
    opted_out_emails)
```

Initially, I think we'd simply check each query individually. In that case, I think it'd be great if we were able to implement these checks as classes similar to those in `src/sqlfluff/rules`. I was thinking a plug-in architecture could be good for this.

Farther in the future, I could imagine doing more sophisticated analysis, such as looking across queries (either CTEs within a single query, or separate queries). In this case, I'm not sure how that would work, but perhaps we'd want to just use SQLFluff as a library, not using the command line. There is _potentially_ a tool idea there as well -- having a tool to output the parse tree in a neutral format like JSON, etc.

Discussion:
barrywhart: This is really exciting! I created the issue for this work, so I am looking forward to creating a plug-in for our company projects.

A few comments responding to your PR description. I will also review the PR later.

>multiple RuleSets

If someone installs multiple plugin packages, or if a single plugin package provides multiple distinct sets of rules, separate rule sets could be a convenient way of enabling or disabling related groups of rules. I can think of rule categories like:
* Formatting
* Correctness
* Security/privacy
* Team or department-specific rule sets based on the common tables/models they own and tend to reuse across projects
* etc.

TBH, if we add the above, could this be useful for the core rule set as well? I have been wondering if we need a way to prevent SQLFluff upgrades being disruptive, when new rules appear and start failing. We could avoid this by grouping them into rule sets with a version number and letting users configure, e.g. `rules=version0.1,version0.2,version0.3,version0.4`. This would exclude new rules added in 0.5 and beyond.

>I added a way for users to include a default configuration for the plugin rules.

Will this also allow users to override the default configuration for the standard, built-in rules? I could perhaps see this being useful, so an organization can have every SQL "project" use the same configuration. For example, all our projects use:

```
dialect = bigquery
templater = jinja
exclude_rules = L029,L031,L032,L034

[sqlfluff:rules:L010]
capitalisation_policy = consistent

[sqlfluff:rules:L014]
capitalisation_policy = consistent

[sqlfluff:rules:L030]
capitalisation_policy = consistent
```

Both these ideas might be at least partly out of scope, which is fine -- just wanted to get some input.
barrywhart: I notice this PR does not link to the related issue. I think it is implementing the idea described in #164.
alanmcruickshank: I've got views on a couple of bits of this:

> Find a better example custom rule (any idea?).

A good example might be to outlaw a particular keyword or method. I could imagine some projects might want to outlaw window functions or maybe outlaw `ORDER BY` clauses. A rule which fails if it finds a particular piece of syntax like that might be a good example.

> I see that the current RuleSet is instantiated with the name "standard" which suggests there would be multiple RuleSets?

Initially I thought that different configurable RuleSets might be a great idea, but the way that sqlfluff has gone since then means we've drifted a long way from the basic code I put in place. I think there are several places in the codebase which are currently incompatible with multiple RuleSets if we did want to use them, so if we did want to use them then I think it's a much bigger conversation which I'm tempted to sidestep entirely for now. I like @barrywhart 's ideas about grouping rules instead - but I think we should deal with that as a seperate issue to this one. For now I think let's assume that plugins apply rules directly to the standard RuleSet.

> I changed the naming convention for rule classes to be Rule_<PluginName: optional>_LXXX so plugin maintainers don't have to worry about rule codes clashing.

I think this is a good idea in general but there's a potential UX issue here which I'd like to cover for. The CLI output and rule filtering code is all by code, and I'd rather discourage plugin writers from reusing codes which are also defined in the core sqlfluff library. Having the plugin name in the classname is a great idea, but I'd rather reserve rules prefixed with `L` for core sqlfluff and then allow plugin writers to prefix their rules with any other letter. If we can enforce this directly in the plugin architecture that would be awesome, but if not then I'm happy to just have it as convention (but I would like the example user-rules to start with a letter other than `L`).

> Find a good way to expose our test tooling to plugin maintainers (so they can test rules through the YAML system) (help?)

Curious for what other people think on this one - but my initial preference would be to copy a little of what `pandas` do, and actually have some test routines as part of the core `sqlfluff` module. This would mean for our test suite, we import some methods from `sqlfluff.test` (or `sqlfluff.testing`?) and these would be the methods which we would be exposing for plugin writers to use. This has the nice upshot of incentivising *us* to have good test coverage for our test routines (meta 🤯 ).

> Resolve the cyclic dependency between config and plugin_manager better (help?)

@dmateusp - on the cycling dependency - I don't quite understand where the issue is happening, could you lay out the dependency tree so we can see where the loop is? In theory, if we make all rules into plugins, then it should be possible to instantiate the core of `sqlfluff` without any rules. Each of the rules should then depend on the core. Anything from the rules required by the core should then by fetched somehow, or queried from them when needed. Does that help at all? I feel like I might be explaining myself badly.
barrywhart: +1 for the idea of having test functionality built into SQLFluff. Note that we've done something a bit similar already with the `diff-quality` support. While that's not exactly "test" functionality, it falls into the closely related area of providing helpful tooling integration in addition to core functionality.
barrywhart: IIUC, the circular dependencies arise from trying to call the plug-in manager objects in the "default" implementations.

If I recall correctly, if possible, don't think in terms of a "default" implementation of the plugins. Think of SQLFluff's implementation as just one of many, and use Pluggy's support for indirectly calling plugins to get both the core rules as well as any additional ones. If you do that, your default implementation won't be **calling** Pluggy in the actual code, but only to declare that "this function is an implementation of hook X".

Does this seem helpful? I know it's kind of vague. I've only used Pluggy a couple of times myself, and it's been over a year since the last time. I think if I spend 2-4 hours digging deeper into this, I can make some more specific suggestions. Let me know how it goes and if I you'd like me to dig in deeper. (I'm totally happy to do it but don't want to steal your "fun"!)
dmateusp: @barrywhart maybe I didn't get what you suggested, I thought that in my current way of approaching the plugin, I was implementing sqlfluff's core logic as "one of many" (as you described).
I tried to remove the `lib` module which contained all of the plugin implementations, by calling `@hookimpl` directly on the functions that I previously had inside the lib but the plugin manager can only register a module or a class (not specific functions as far as I understand), so I ended up in the same place.

I just don't understand how to avoid the `implementation module -> plugin_manager -> implementation_module` while keeping a sane module structure.

The only way I managed it so far is by creating a plugin class where we import the different core implementations (getting default configs, finding rules etc..), and use deferred imports in the plugin class methods (a hack at best!) .

It would be great if you could have a look :) 
barrywhart: Sure, I'll take a look to try and make more concrete suggestions. My previous advice was super vague.

I think my initial focus will be on trying to break the `plugin_manager -> implementation_module` part of the cycle.
barrywhart: I'm looking over my last project that used Pluggy -- a project which added plugin support to the `diff-quality` project, which SQLFluff in turn implements in order to support generating `diff-quality` reports on `.sql` files.

I'm not providing any specific advice or suggestions in this comment, but I'll be using this as a reference to see how the PR differs from this, but I'm sharing this for reference, in case it's helpful. I'll follow up later with specific suggestions.

High-level examples (from https://github.com/Bachmann1234/diff_cover/pull/128):

# Top-level module of the project...

`diff_cover/__init__.py`:

```
# Other packages that implement diff_cover plugins use this.
hookimpl = pluggy.HookimplMarker('diff_cover')
```

# Core code defining the "APIs" plugins may implement...

`diff_cover/hookspecs.py`:

```
import pluggy

hookspec = pluggy.HookspecMarker("diff_cover")


@hookspec
def diff_cover_report_quality():
    """
    <<DESCRIBE THE EXPECTED BEHAVIOR OF AN IMPLEMENTATION.>>
    """
```

# Core code loading, enumerating, working with plugins...

`diff_cover/diff_quality_tool.py`:

```
        pm = pluggy.PluginManager('diff_cover')
        pm.add_hookspecs(hookspecs)
        pm.load_setuptools_entrypoints('diff_cover')
        for hookimpl in pm.hook.diff_cover_report_quality.get_hookimpls():
            result = hookimpl.function()
```

# External plug-in code implementing a hook API:

`setup.py`:

```
    setup(
        ...
        entry_points={
            'diff_cover': [
                'sqlfluff = my_package.my_plugin'
            ],
        },
        ...
    )
```

`my_package/my_plugin.py`:

```
    import diff_cover

    @diff_cover.hookimpl
    def diff_cover_report_quality():
        ... DO ALL THE THINGS ...
```
dmateusp: Oops! This should have been removed, sometimes I use it to print things

On Thu 11 Feb 2021, 18:45 Barry Hart, <notifications@github.com> wrote:

> *@barrywhart* commented on this pull request.
> ------------------------------
>
> In src/sqlfluff/core/rules/base.py
> <https://github.com/sqlfluff/sqlfluff/pull/705#discussion_r574741816>:
>
> > @@ -455,15 +456,15 @@ def _validate_config_options(self, config, rule=None):
>          Config options can also be checked for a specific rule e.g L010.
>          """
>          rule_config = config.get_section("rules")
> -
> +        from icecream import ic
>
> What's this line? While there *is* a package called icecream, it's not
> listed in the requirements files.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/sqlfluff/sqlfluff/pull/705#pullrequestreview-588875838>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACDMIMUDGXG3S3JIFYKBWQDS6QQTNANCNFSM4WOG7HDA>
> .
>

barrywhart: @dmateusp: I think I have addressed the dependency cycle. Some tests are still failing, but I think the basic plumbing is working. In order to let us review and discuss the proposed changes to your PR, I created a PR against your PR. 😄

Here it is: https://github.com/dmateusp/sqlfluff/pull/1
barrywhart: 🍿🙏🏽 (Excitement!)
dmateusp: ## Plugin - Rule naming

@alanmcruickshank 

> I think this is a good idea in general but there's a potential UX issue here which I'd like to cover for. The CLI output and rule filtering code is all by code, and I'd rather discourage plugin writers from reusing codes which are also defined in the core sqlfluff library. Having the plugin name in the classname is a great idea, but I'd rather reserve rules prefixed with L for core sqlfluff and then allow plugin writers to prefix their rules with any other letter. If we can enforce this directly in the plugin architecture that would be awesome, but if not then I'm happy to just have it as convention (but I would like the example user-rules to start with a letter other than L).

I wanted to talk about this more. So in the current set up `L001` in a plugin should not clash with `L001` in core. This is because in the case of a plugin, I'm including the plugin name in the code e.g. `Example_L001` rather than `L001`.

In practice it looks like:
```
== [order_by.sql] FAIL
L:   3 | P:  10 | Example_L001 | Column `bar` not allowed in ORDER BY.
```

To me, the letter in front of the number stands for the "type" of rule. Right now we have only "L" for linting, and "P" for parsing AFAIK. But I can imagine @barrywhart could implement his security plugins as "S" for example?

# Testing

I added a `sqlfluff.testing` module, and used it in the example plugin. It actually surface an issue in my code, so I guess it works as expected 😉 

dmateusp: FYI I still need to add documentation
codecov[bot]: # [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=h1) Report
> Merging [#705](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=desc) (c922225) into [master](https://codecov.io/gh/sqlfluff/sqlfluff/commit/71facce92cfd47cbe1e3eb9214ff29a888a7718a?el=desc) (71facce) will **decrease** coverage by `0.00%`.
> The diff coverage is `94.93%`.

[![Impacted file tree graph](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/graphs/tree.svg?width=650&height=150&src=pr&token=maM8kLgCmn)](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #705      +/-   ##
==========================================
- Coverage   95.51%   95.51%   -0.01%     
==========================================
  Files          92       97       +5     
  Lines        5980     6103     +123     
==========================================
+ Hits         5712     5829     +117     
- Misses        268      274       +6     
```

| Flag | Coverage Δ | |
|---|---|---|
| dbt017-py38 | `56.59% <78.48%> (+0.35%)` | :arrow_up: |
| dbt018-py38 | `56.62% <78.48%> (+0.35%)` | :arrow_up: |
| py36 | `94.65% <94.83%> (+<0.01%)` | :arrow_up: |
| py37 | `94.65% <94.83%> (+<0.01%)` | :arrow_up: |
| py38 | `94.61% <94.93%> (+0.01%)` | :arrow_up: |
| py39 | `94.61% <94.93%> (+0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/sqlfluff/core/rules/std/\_\_init\_\_.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcnVsZXMvc3RkL19faW5pdF9fLnB5) | `88.23% <85.71%> (+0.73%)` | :arrow_up: |
| [src/sqlfluff/testing/rules.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL3Rlc3RpbmcvcnVsZXMucHk=) | `91.04% <91.04%> (ø)` | |
| [src/sqlfluff/cli/commands.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NsaS9jb21tYW5kcy5weQ==) | `91.69% <100.00%> (+0.03%)` | :arrow_up: |
| [src/sqlfluff/core/config.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvY29uZmlnLnB5) | `96.25% <100.00%> (+0.03%)` | :arrow_up: |
| [src/sqlfluff/core/parser/grammar/base.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGFyc2VyL2dyYW1tYXIvYmFzZS5weQ==) | `93.33% <100.00%> (ø)` | |
| [src/sqlfluff/core/parser/lexer.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGFyc2VyL2xleGVyLnB5) | `97.74% <100.00%> (ø)` | |
| [src/sqlfluff/core/plugin/\_\_init\_\_.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGx1Z2luL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [src/sqlfluff/core/plugin/hookspecs.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGx1Z2luL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/sqlfluff/core/plugin/host.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGx1Z2luL2hvc3QucHk=) | `100.00% <100.00%> (ø)` | |
| [src/sqlfluff/core/plugin/lib.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NvcmUvcGx1Z2luL2xpYi5weQ==) | `100.00% <100.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=footer). Last update [71facce...c922225](https://codecov.io/gh/sqlfluff/sqlfluff/pull/705?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

dmateusp: FYI I hit many conflicts with the recent merges, trying to figure it out
dmateusp: Amazing, thanks for the 👍 @barrywhart and everyone else for chiming in :) I'm going to merge the PR now

Patch:
diff --git a/docs/source/developingplugins.rst b/docs/source/developingplugins.rst
new file mode 100644
index 000000000..aadf4579c
--- /dev/null
+++ b/docs/source/developingplugins.rst
@@ -0,0 +1,56 @@
+.. _developingpluginsref:
+
+Developing Plugins
+==================
+
+*SQLFluff* is extensible through "plugins". We use the `pluggy library`_
+to make linting Rules pluggable, which enable users to implement rules that
+are just too "organization specific" to be shared, or too platform specific
+to be included in the core library.
+
+.. _`pluggy library`: https://pluggy.readthedocs.io/en/latest/
+
+Creating a plugin
+-----------------
+
+We have an example plugin in `sqlfluff/plugins/sqlfluff-plugin-example`_
+which you can use as a template.
+
+Few things to note about plugins:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Right now, only Rules can be added through plugins. Each plugin can
+implement multiple Rules.
+
+The name of a plugin should start with *"sqlfluff-plugin"* to be valid.
+
+A plugin may need to include a default configuration if its rules
+are configurable: use plugin default configurations **only for that reason**!
+We advise against overwriting core configurations by using a default
+plugin configuration, as there is no mechanism in place to enforce precendence
+between the core library configs and plugin configs,
+and multiple plugins could clash.
+
+A plugin Rule class name should have the structure:
+"Rule_PluginName_L000". The 'L' can be any letter
+and is meant to categorize rules; you could use the
+letter 'S' to denote rules that enforce security checks
+for example.
+
+A plugin Rule code includes the PluginName,
+so a rule "Rule_L000" in core will have code "L000",
+while "Rule_PluginName_L000" will have code "PluginName_L000".
+Codes are used to display errors, they are also used as configuration keys.
+
+We make it easy for plugin developers to test their rules by
+exposing a testing library in *sqlfluff.testing*.
+
+.. _`sqlfluff/plugins/sqlfluff-plugin-example`: https://github.com/sqlfluff/sqlfluff/tree/master/plugins/sqlfluff-plugin-example
+
+Giving feedback
+---------------
+
+Would you like to have other parts of *SQLFluff* be "pluggable"?
+Tell us about it in a `GitHub issue`_ 😄.
+
+.. _`GitHub issue`: https://github.com/sqlfluff/sqlfluff/issues/new?assignees=&labels=enhancement&template=enhancement.md
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 964ddaa53..df8ebd97c 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -59,6 +59,7 @@ Contents
    architecture
    cli
    api
+   developingplugins
    inthewild
    jointhecommunity
 
diff --git a/mypy.ini b/mypy.ini
index 6ad5b1b9a..5b03e6f40 100644
--- a/mypy.ini
+++ b/mypy.ini
@@ -26,3 +26,6 @@ ignore_missing_imports = True
 
 [mypy-dbt.*]
 ignore_missing_imports = True
+
+[mypy-pluggy.*]
+ignore_missing_imports = True
diff --git a/plugins/sqlfluff-plugin-example/MANIFEST.in b/plugins/sqlfluff-plugin-example/MANIFEST.in
new file mode 100644
index 000000000..1eb895549
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/MANIFEST.in
@@ -0,0 +1,1 @@
+include src/example/plugin_default_config.cfg
diff --git a/plugins/sqlfluff-plugin-example/setup.py b/plugins/sqlfluff-plugin-example/setup.py
new file mode 100644
index 000000000..169ba227a
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/setup.py
@@ -0,0 +1,15 @@
+"""Setup file for example plugin."""
+from setuptools import setup
+
+setup(
+    name="sqlfluff-plugin-example",
+    include_package_data=True,
+    package_dir={"": "src"},
+    packages=[
+        "example",
+    ],
+    # TODO: Change me to the first release
+    # that includes the pluggy work.
+    install_requires="sqlfluff>=0.4.0a3",
+    entry_points={"sqlfluff": ["example = example.rules"]},
+)
diff --git a/plugins/sqlfluff-plugin-example/src/example/__init__.py b/plugins/sqlfluff-plugin-example/src/example/__init__.py
new file mode 100644
index 000000000..c651355cd
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/src/example/__init__.py
@@ -0,0 +1,1 @@
+"""Example sqlfluff plugin."""
diff --git a/plugins/sqlfluff-plugin-example/src/example/plugin_default_config.cfg b/plugins/sqlfluff-plugin-example/src/example/plugin_default_config.cfg
new file mode 100644
index 000000000..b0902b409
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/src/example/plugin_default_config.cfg
@@ -0,0 +1,2 @@
+[sqlfluff:rules:Example_L001]
+forbidden_columns = bar, baaz
diff --git a/plugins/sqlfluff-plugin-example/src/example/rules.py b/plugins/sqlfluff-plugin-example/src/example/rules.py
new file mode 100644
index 000000000..b341bc2b6
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/src/example/rules.py
@@ -0,0 +1,87 @@
+"""An example of a custom rule implemented through the plugin system."""
+
+from sqlfluff.core.plugin import hookimpl
+from sqlfluff.core.rules.base import (
+    BaseCrawler,
+    LintResult,
+)
+from sqlfluff.core.rules.doc_decorators import (
+    document_fix_compatible,
+    document_configuration,
+)
+from typing import List
+import os.path
+from sqlfluff.core.config import ConfigLoader
+
+
+@hookimpl
+def get_rules() -> List[BaseCrawler]:
+    """Get plugin rules."""
+    return [Rule_Example_L001]
+
+
+@hookimpl
+def load_default_config() -> dict:
+    """Loads the default configuration for the plugin."""
+    return ConfigLoader.get_global().load_default_config_file(
+        file_dir=os.path.dirname(__file__),
+        file_name="plugin_default_config.cfg",
+    )
+
+
+@hookimpl
+def get_configs_info() -> dict:
+    """Get rule config validations and descriptions."""
+    return {
+        "forbidden_columns": {"definition": "A list of column to forbid"},
+    }
+
+
+@document_fix_compatible
+@document_configuration
+class Rule_Example_L001(BaseCrawler):
+    """ORDER BY on these columns is forbidden!
+
+    | **Anti-pattern**
+    | Using ORDER BY one some forbidden columns.
+
+    .. code-block::
+
+        SELECT *
+        FROM foo
+        ORDER BY
+            bar,
+            baz
+
+    | **Best practice**
+    | Do not order by these columns.
+
+    .. code-block::
+
+        SELECT *
+        FROM foo
+        ORDER BY bar
+    """
+
+    config_keywords = ["forbidden_columns"]
+
+    def __init__(self, *args, **kwargs):
+        """Overwrite __init__ to set config."""
+        super().__init__(*args, **kwargs)
+        self.forbidden_columns = [
+            col.strip() for col in self.forbidden_columns.split(",")
+        ]
+
+    def _eval(self, segment, raw_stack, **kwargs):
+        """We should not use ORDER BY."""
+        if segment.is_type("orderby_clause"):
+            for seg in segment.segments:
+                col_name = seg.raw.lower()
+                if (
+                    seg.is_type("column_reference")
+                    and col_name in self.forbidden_columns
+                ):
+                    return LintResult(
+                        anchor=seg,
+                        description=f"Column `{col_name}` not allowed in ORDER BY.",
+                    )
diff --git a/setup.py b/setup.py
index 2aa79d9b4..22d6ff210 100644
--- a/setup.py
+++ b/setup.py
@@ -109,5 +109,6 @@ setup(
             "sqlfluff = sqlfluff.cli.commands:cli",
         ],
         "diff_cover": ["sqlfluff = sqlfluff.diff_quality_plugin"],
+        "sqlfluff": ["sqlfluff = sqlfluff.core.plugin.lib"],
     },
 )
diff --git a/src/sqlfluff/core/config.py b/src/sqlfluff/core/config.py
index 9a4d2e00b..f06701b5b 100644
--- a/src/sqlfluff/core/config.py
+++ b/src/sqlfluff/core/config.py
@@ -7,6 +7,7 @@ import sys
 import configparser
 from typing import Dict, List, Tuple, Any, Optional, Union, Iterable
 from pathlib import Path
+from sqlfluff.core.plugin.host import get_plugin_manager
 
 import appdirs
 
@@ -222,11 +223,9 @@ class ConfigLoader:
             r[n] = v
         return ctx
 
-    def load_default_config_file(self) -> dict:
+    def load_default_config_file(self, file_dir: str, file_name: str) -> dict:
         """Load the default config file."""
-        elems = self._get_config_elems_from_file(
-            os.path.join(os.path.dirname(__file__), "default_config.cfg")
-        )
+        elems = self._get_config_elems_from_file(os.path.join(file_dir, file_name))
         return self._incorporate_vals({}, elems)
 
     def load_config_at_path(self, path: str) -> dict:
@@ -338,7 +337,7 @@ class FluffConfig:
         self, configs: Optional[dict] = None, overrides: Optional[dict] = None
     ):
         self._overrides = overrides  # We only store this for child configs
-        defaults = ConfigLoader.get_global().load_default_config_file()
+        defaults = nested_combine(*get_plugin_manager().hook.load_default_config())
         self._configs = nested_combine(
             defaults, configs or {"core": {}}, {"core": overrides or {}}
         )
diff --git a/src/sqlfluff/core/plugin/__init__.py b/src/sqlfluff/core/plugin/__init__.py
new file mode 100644
index 000000000..9e85a1a30
--- /dev/null
+++ b/src/sqlfluff/core/plugin/__init__.py
@@ -0,0 +1,6 @@
+"""Marker to be imported and used in plugins (and for own implementations)."""
+import pluggy
+
+project_name = "sqlfluff"
+plugin_base_name = f"{project_name}-plugin"
+hookimpl = pluggy.HookimplMarker(plugin_base_name)
diff --git a/src/sqlfluff/core/plugin/hookspecs.py b/src/sqlfluff/core/plugin/hookspecs.py
new file mode 100644
index 000000000..5fb480cfe
--- /dev/null
+++ b/src/sqlfluff/core/plugin/hookspecs.py
@@ -0,0 +1,21 @@
+"""Defines the specification to implement a plugin."""
+
+import pluggy
+from sqlfluff.core.plugin import plugin_base_name
+
+hookspec = pluggy.HookspecMarker(plugin_base_name)
+
+
+class PluginSpec:
+    """Defines the method signatures for plugin implementations."""
+
+    @hookspec
+    def get_rules(self):
+        """Get plugin rules."""
+
+    def load_default_config(self) -> dict:
+        """Loads the default configuration for the plugin."""
+
+    @hookspec
+    def get_configs_info(self) -> dict:
+        """Get rule config validations and descriptions."""
diff --git a/src/sqlfluff/core/plugin/host.py b/src/sqlfluff/core/plugin/host.py
new file mode 100644
index 000000000..19ac819cb
--- /dev/null
+++ b/src/sqlfluff/core/plugin/host.py
@@ -0,0 +1,13 @@
+"""Defines the plugin manager getter."""
+import pluggy
+
+from sqlfluff.core.plugin.hookspecs import PluginSpec
+from sqlfluff.core.plugin import plugin_base_name, project_name
+
+
+def get_plugin_manager() -> pluggy.PluginManager:
+    """Initializes the PluginManager."""
+    pm = pluggy.PluginManager(plugin_base_name)
+    pm.add_hookspecs(PluginSpec)
+    pm.load_setuptools_entrypoints(project_name)
+    return pm
diff --git a/src/sqlfluff/core/plugin/lib.py b/src/sqlfluff/core/plugin/lib.py
new file mode 100644
index 000000000..37974012d
--- /dev/null
+++ b/src/sqlfluff/core/plugin/lib.py
@@ -0,0 +1,28 @@
+"""Base implementation for the plugin."""
+
+import os.path
+from sqlfluff.core.config import ConfigLoader
+from sqlfluff.core.plugin import hookimpl
+from sqlfluff.core.rules.config_info import STANDARD_CONFIG_INFO_DICT
+from sqlfluff.core.rules.std import get_rules_from_path
+
+
+@hookimpl
+def get_rules():
+    """Get plugin rules."""
+    return get_rules_from_path()
+
+
+@hookimpl
+def load_default_config() -> dict:
+    """Loads the default configuration for the plugin."""
+    return ConfigLoader.get_global().load_default_config_file(
+        file_dir=os.path.join(os.path.dirname(os.path.dirname(__file__))),
+        file_name="default_config.cfg",
+    )
+
+
+@hookimpl
+def get_configs_info() -> dict:
+    """Get rule config validations and descriptions."""
+    return STANDARD_CONFIG_INFO_DICT
diff --git a/src/sqlfluff/core/rules/__init__.py b/src/sqlfluff/core/rules/__init__.py
index e26327878..989ba7672 100644
--- a/src/sqlfluff/core/rules/__init__.py
+++ b/src/sqlfluff/core/rules/__init__.py
@@ -2,16 +2,14 @@
 
 from sqlfluff.core.rules.base import RuleSet
 from sqlfluff.core.rules.config_info import STANDARD_CONFIG_INFO_DICT
-from sqlfluff.core.rules.std import (
-    rules as std_rules,
-)  # Import the standard ruleset list
-
+from sqlfluff.core.plugin.host import get_plugin_manager
 
 std_rule_set = RuleSet(name="standard", config_info=STANDARD_CONFIG_INFO_DICT)
 
-# Iterate through the std rules list and register each rule with the std_rule_set
-for rule in std_rules:
-    std_rule_set.register(rule)
+# Iterate through the rules list and register each rule with the std_rule_set
+for plugin_rules in get_plugin_manager().hook.get_rules():
+    for rule in plugin_rules:
+        std_rule_set.register(rule)
 
 
 def get_ruleset(name: str = "standard") -> RuleSet:
diff --git a/src/sqlfluff/core/rules/base.py b/src/sqlfluff/core/rules/base.py
index d9ff5226a..4df07b824 100644
--- a/src/sqlfluff/core/rules/base.py
+++ b/src/sqlfluff/core/rules/base.py
@@ -16,6 +16,7 @@ missing.
 
 import copy
 import logging
+import re
 from collections import namedtuple
 
 from sqlfluff.core.parser import RawSegment, KeywordSegment, BaseSegment, SymbolSegment
@@ -463,15 +464,18 @@ class RuleSet:
         Config options can also be checked for a specific rule e.g L010.
         """
         rule_config = config.get_section("rules")
-
         for config_name, info_dict in self.config_info.items():
             config_option = (
                 rule_config.get(config_name)
                 if not rule
                 else rule_config.get(rule).get(config_name)
             )
-            valid_options = info_dict["validation"]
-            if config_option not in valid_options and config_option is not None:
+            valid_options = info_dict.get("validation")
+            if (
+                valid_options
+                and config_option not in valid_options
+                and config_option is not None
+            ):
                 raise ValueError(
                     (
                         "Invalid option '{0}' for {1} configuration. Must be one of {2}"
@@ -482,7 +486,21 @@ class RuleSet:
                     )
                 )
 
-    def register(self, cls):
+    @property
+    def valid_rule_name_regex(self):
+        """Defines the accepted pattern for rule names.
+
+        The first group captures the plugin name (optional), which
+        must be capitalized.
+        The second group captures the rule code.
+
+        Examples of valid rule names:
+        * Rule_PluginName_L001
+        * Rule_L001
+        """
+        return re.compile(r"Rule_?([A-Z]{1}[a-zA-Z]+)?_([A-Z][0-9]{3})")
+
+    def register(self, cls, plugin=None):
         """Decorate a class with this to add it to the ruleset.
 
         .. code-block:: python
@@ -502,19 +520,24 @@ class RuleSet:
         :exc:`ValueError`.
 
         """
-        elems = cls.__name__.split("_")
+        rule_name_match = self.valid_rule_name_regex.match(cls.__name__)
         # Validate the name
-        if len(elems) != 2 or elems[0] != "Rule" or len(elems[1]) != 4:
+        if not rule_name_match:
             raise ValueError(
                 (
-                    "Tried to register rule on set {0!r} with unexpected " "format: {1}"
+                    "Tried to register rule on set {0!r} with unexpected "
+                    "format: {1}, format should be: Rule_PluginName_L123 (for plugins) "
+                    "or Rule_L123 (for core rules)."
                 ).format(self.name, cls.__name__)
             )
 
-        code = elems[1]
+        plugin_name, code = rule_name_match.groups()
         # If the docstring is multiline, then we extract just summary.
         description = cls.__doc__.split("\n")[0]
 
+        if plugin_name:
+            code = f"{plugin_name}_{code}"
+
         # Keep track of the *class* in the register. Don't instantiate yet.
         if code in self._register:
             raise ValueError(
diff --git a/src/sqlfluff/core/rules/config_info.py b/src/sqlfluff/core/rules/config_info.py
index 42d436180..70d4044dd 100644
--- a/src/sqlfluff/core/rules/config_info.py
+++ b/src/sqlfluff/core/rules/config_info.py
@@ -7,6 +7,8 @@ This mapping is used to validate rule config inputs, as well
 as document rule configuration.
 """
 
+from sqlfluff.core.plugin.host import get_plugin_manager
+
 STANDARD_CONFIG_INFO_DICT = {
     "tab_space_size": {
         "validation": range(100),
@@ -76,3 +78,12 @@ STANDARD_CONFIG_INFO_DICT = {
         "definition": "Which clauses should be linted for subqueries",
     },
 }
+
+
+def get_config_info() -> dict:
+    """Gets the config from core sqlfluff and sqlfluff plugins and merges them."""
+    plugin_manager = get_plugin_manager()
+    configs_info = plugin_manager.hook.get_configs_info()
+    return {
+        k: v for config_info_dict in configs_info for k, v in config_info_dict.items()
+    }
diff --git a/src/sqlfluff/core/rules/doc_decorators.py b/src/sqlfluff/core/rules/doc_decorators.py
index 7c54571f9..07c9e2ccc 100644
--- a/src/sqlfluff/core/rules/doc_decorators.py
+++ b/src/sqlfluff/core/rules/doc_decorators.py
@@ -1,6 +1,6 @@
 """A collection of decorators to modify rule docstrings for Sphinx."""
 
-from sqlfluff.core.rules.config_info import STANDARD_CONFIG_INFO_DICT
+from sqlfluff.core.rules.config_info import get_config_info
 from sqlfluff.core.rules.base import rules_logger  # noqa
 
 
@@ -28,7 +28,7 @@ def document_configuration(cls, ruleset="std"):
     options in the docs, from a single source of truth.
     """
     if ruleset == "std":
-        config_info = STANDARD_CONFIG_INFO_DICT
+        config_info = get_config_info()
     else:
         raise (
             NotImplementedError(
@@ -47,9 +47,11 @@ def document_configuration(cls, ruleset="std"):
                         keyword, cls.__name__
                     )
                 )
-            config_doc += "\n    |     `{0}`: {1}. Must be one of {2}.".format(
-                keyword, info_dict["definition"], info_dict["validation"]
+            config_doc += "\n    |     `{0}`: {1}.".format(
+                keyword, info_dict["definition"]
             )
+            if "validation" in info_dict:
+                config_doc += " Must be one of {0}.".format(info_dict["validation"])
             config_doc += "\n    |"
     except AttributeError:
         rules_logger.info("No config_keywords defined for {0}".format(cls.__name__))
diff --git a/src/sqlfluff/core/rules/std/__init__.py b/src/sqlfluff/core/rules/std/__init__.py
index d4eb05e18..f276ac63d 100644
--- a/src/sqlfluff/core/rules/std/__init__.py
+++ b/src/sqlfluff/core/rules/std/__init__.py
@@ -6,32 +6,40 @@ import os
 from importlib import import_module
 from glob import glob
 
-# All rule files are expected in the format of L*.py
-rules_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "L*.py")
 
-# Create a rules dictionary for importing in sqlfluff/src/sqlfluff/core/rules/__init__.py
-rules = []
+def get_rules_from_path(
+    # All rule files are expected in the format of L*.py
+    rules_path=os.path.join(os.path.abspath(os.path.dirname(__file__)), "L*.py"),
+    base_module="sqlfluff.core.rules.std",
+):
+    """Reads all of the Rule classes from a path into a list."""
+    # Create a rules dictionary for importing in sqlfluff/src/sqlfluff/core/rules/__init__.py
+    rules = []
 
-# Sphinx effectively runs an import * from this module in rules.rst, so initialise
-# __all__ with an empty list before we populate it with the rule names.
-__all__ = []
+    # Sphinx effectively runs an import * from this module in rules.rst, so initialise
+    # __all__ with an empty list before we populate it with the rule names.
+    __all__ = []
 
-for module in sorted(glob(rules_path)):
-    # Manipulate the module path to extract the filename without the .py
-    rule_id = os.path.splitext(os.path.basename(module))[0]
-    # All rule classes are expected in the format of Rule_L*
-    rule_class_name = f"Rule_{rule_id}"
-    try:
-        rule_class = getattr(
-            import_module(f"sqlfluff.core.rules.std.{rule_id}"), rule_class_name
-        )
-    except AttributeError:
-        raise (AttributeError("Rule classes must be named in the format of L*."))
-    # Add the rules to the rules dictionary for sqlfluff/src/sqlfluff/core/rules/__init__.py
-    rules.append(rule_class)
-    # Add the rule_classes to the module namespace with globals() so that they can
-    # be found by Sphinx automodule documentation in rules.rst
-    # The result is the same as declaring the classes in this file.
-    globals()[rule_class_name] = rule_class
-    # Add the rule class names to __all__ for Sphinx automodule discovery
-    __all__.append(rule_class_name)
+    for module in sorted(glob(rules_path)):
+        # Manipulate the module path to extract the filename without the .py
+        rule_id = os.path.splitext(os.path.basename(module))[0]
+        # All rule classes are expected in the format of Rule_L*
+        rule_class_name = f"Rule_{rule_id}"
+        try:
+            rule_class = getattr(
+                import_module(f"{base_module}.{rule_id}"), rule_class_name
+            )
+        except AttributeError as e:
+            raise AttributeError(
+                "Rule classes must be named in the format of L*."
+            ) from e
+        # Add the rules to the rules dictionary for sqlfluff/src/sqlfluff/core/rules/__init__.py
+        rules.append(rule_class)
+        # Add the rule_classes to the module namespace with globals() so that they can
+        # be found by Sphinx automodule documentation in rules.rst
+        # The result is the same as declaring the classes in this file.
+        globals()[rule_class_name] = rule_class
+        # Add the rule class names to __all__ for Sphinx automodule discovery
+        __all__.append(rule_class_name)
+
+    return rules
diff --git a/tox.ini b/tox.ini
index 88d15faa6..cb7185821 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,6 +12,8 @@ deps =
     -Urrequirements_dev.txt
     # Requests is required for the util script
     requests
+    # Add the example plugin
+    plugins/sqlfluff-plugin-example
     # Define dbt versions
     dbt017: dbt==0.17.2
     dbt018: dbt==0.18.1


Test patch:
diff --git a/plugins/sqlfluff-plugin-example/test/rules/rule_test_cases_test.py b/plugins/sqlfluff-plugin-example/test/rules/rule_test_cases_test.py
new file mode 100644
index 000000000..d6b936367
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/test/rules/rule_test_cases_test.py
@@ -0,0 +1,16 @@
+"""Runs the rule test cases."""
+import os
+import pytest
+from sqlfluff.testing.rules import load_test_cases, rules__test_helper
+
+ids, test_cases = load_test_cases(
+    test_cases_path=os.path.join(
+        os.path.abspath(os.path.dirname(__file__)), "test_cases", "*.yml"
+    )
+)
+
+
+@pytest.mark.parametrize("test_case", test_cases, ids=ids)
+def test__rule_test_case(test_case):
+    """Run the tests."""
+    rules__test_helper(test_case)
diff --git a/plugins/sqlfluff-plugin-example/test/rules/test_cases/Rule_Example_L001.yml b/plugins/sqlfluff-plugin-example/test/rules/test_cases/Rule_Example_L001.yml
new file mode 100644
index 000000000..b4ec9e24e
--- /dev/null
+++ b/plugins/sqlfluff-plugin-example/test/rules/test_cases/Rule_Example_L001.yml
@@ -0,0 +1,26 @@
+rule: Example_L001
+
+no_forbidden_col_used:
+  pass_str: |
+    select
+      a,
+      sum(b)
+    from tbl
+    group by a
+    order by a
+
+no_order_by_used:
+  pass_str: |
+    select
+      a,
+      b,
+      c
+    from tbl
+
+forbidden_col_used:
+  fail_str: |
+    select
+      bar,
+      baz
+    from tbl
+    order by bar
diff --git a/src/sqlfluff/testing/__init__.py b/src/sqlfluff/testing/__init__.py
new file mode 100644
index 000000000..c418a4a8a
--- /dev/null
+++ b/src/sqlfluff/testing/__init__.py
@@ -0,0 +1,1 @@
+"""Testing utils we want to expose for usage by plugins."""
diff --git a/src/sqlfluff/testing/rules.py b/src/sqlfluff/testing/rules.py
new file mode 100644
index 000000000..77d0d5285
--- /dev/null
+++ b/src/sqlfluff/testing/rules.py
@@ -0,0 +1,122 @@
+"""Testing utils for rule plugins."""
+from sqlfluff.core import Linter
+from sqlfluff.core.errors import SQLParseError
+from sqlfluff.core.rules import get_ruleset
+from sqlfluff.core.config import FluffConfig
+from typing import Tuple, List, NamedTuple, Optional
+from glob import glob
+
+import pytest
+import oyaml as yaml
+
+
+class RuleTestCase(NamedTuple):
+    """Used like a dataclass by rule tests."""
+
+    rule: Optional[str] = None
+    desc: Optional[str] = None
+    pass_str: Optional[str] = None
+    fail_str: Optional[str] = None
+    fix_str: Optional[str] = None
+    configs: Optional[dict] = None
+    skip: Optional[str] = None
+
+
+def load_test_cases(
+    test_cases_path: str,
+) -> Tuple[List[str], List[RuleTestCase]]:
+    """Load rule test cases from YAML files."""
+    ids = []
+    test_cases = []
+
+    for path in sorted(glob(test_cases_path)):
+        with open(path) as f:
+            raw = f.read()
+
+        y = yaml.safe_load(raw)
+
+        rule = y.pop("rule")
+        ids.extend([rule + "_" + t for t in y])
+        test_cases.extend([RuleTestCase(rule=rule, **v) for k, v in y.items()])
+
+    return ids, test_cases
+
+
+def get_rule_from_set(code, config):
+    """Fetch a rule from the rule set."""
+    for r in get_ruleset().get_rulelist(config=config):
+        if r.code == code:
+            return r
+    raise ValueError("{0!r} not in {1!r}".format(code, get_ruleset()))
+
+
+def assert_rule_fail_in_sql(code, sql, configs=None):
+    """Assert that a given rule does fail on the given sql."""
+    # Set up the config to only use the rule we are testing.
+    cfg = FluffConfig(configs=configs, overrides={"rules": code})
+    # Lint it using the current config (while in fix mode)
+    linted = Linter(config=cfg).lint_string(sql, fix=True)
+    lerrs = linted.get_violations()
+    print("Errors Found: {0}".format(lerrs))
+    parse_errors = list(filter(lambda v: type(v) == SQLParseError, lerrs))
+    if parse_errors:
+        pytest.fail(f"Found the following parse errors in test case: {parse_errors}")
+    if not any(v.rule.code == code for v in lerrs):
+        pytest.fail(
+            "No {0} failures found in query which should fail.".format(code),
+            pytrace=False,
+        )
+    # The query should already have been fixed if possible so just return the raw.
+    return linted.tree.raw
+
+
+def assert_rule_pass_in_sql(code, sql, configs=None):
+    """Assert that a given rule doesn't fail on the given sql."""
+    # Configs allows overrides if we want to use them.
+    cfg = FluffConfig(configs=configs)
+    r = get_rule_from_set(code, config=cfg)
+    parsed = Linter(config=cfg).parse_string(sql)
+    if parsed.violations:
+        pytest.fail(parsed.violations[0].desc() + "\n" + parsed.tree.stringify())
+    print("Parsed:\n {0}".format(parsed.tree.stringify()))
+    lerrs, _, _, _ = r.crawl(parsed.tree, dialect=cfg.get("dialect_obj"))
+    print("Errors Found: {0}".format(lerrs))
+    if any(v.rule.code == code for v in lerrs):
+        pytest.fail(
+            "Found {0} failures in query which should pass.".format(code), pytrace=False
+        )
+
+
+def assert_rule_raises_violations_in_file(rule, fpath, violations, fluff_config):
+    """Assert that a given rule raises given errors in specific positions of a file."""
+    lntr = Linter(config=fluff_config)
+    lnt = lntr.lint_path(fpath)
+    # Reformat the test data to match the format we're expecting. We use
+    # sets because we really don't care about order and if one is missing,
+    # we don't care about the orders of the correct ones.
+    assert set(lnt.check_tuples()) == {(rule, v[0], v[1]) for v in violations}
+
+
+def rules__test_helper(test_case):
+    """Test that a rule passes/fails on a set of test_cases.
+
+    Optionally, also test the fixed string if provided in the test case.
+    """
+    if test_case.skip:
+        pytest.skip(test_case.skip)
+
+    if test_case.pass_str:
+        assert_rule_pass_in_sql(
+            test_case.rule,
+            test_case.pass_str,
+            configs=test_case.configs,
+        )
+    if test_case.fail_str:
+        res = assert_rule_fail_in_sql(
+            test_case.rule,
+            test_case.fail_str,
+            configs=test_case.configs,
+        )
+        # If a `fixed` value is provided then check it matches
+        if test_case.fix_str:
+            assert res == test_case.fix_str
diff --git a/test/core/plugin_test.py b/test/core/plugin_test.py
new file mode 100644
index 000000000..58457aab2
--- /dev/null
+++ b/test/core/plugin_test.py
@@ -0,0 +1,31 @@
+"""Plugin related tests."""
+from sqlfluff.core.plugin.host import get_plugin_manager
+from sqlfluff.core.config import FluffConfig
+
+
+def test__plugin_manager_registers_example_plugin():
+    """Test that the example plugin is registered."""
+    plugin_manager = get_plugin_manager()
+    # The plugin import order is non-deterministic
+    assert sorted(
+        [plugin_module.__name__ for plugin_module in plugin_manager.get_plugins()]
+    ) == [
+        "example.rules",
+        "sqlfluff.core.plugin.lib",
+    ]
+
+
+def test__plugin_example_rules_returned():
+    """Test that the example rules from the plugin are returned."""
+    plugin_manager = get_plugin_manager()
+    # The plugin import order is non-deterministic
+    assert "Rule_Example_L001" in [
+        rule.__name__ for rules in plugin_manager.hook.get_rules() for rule in rules
+    ]
+
+
+def test__plugin_default_config_read():
+    """Test that the example plugin default config is merged into FluffConfig."""
+    fluff_config = FluffConfig()
+    # The plugin import order is non-deterministic
+    assert "forbidden_columns" in fluff_config._configs["rules"]["Example_L001"]
diff --git a/test/core/rules/rule_test_cases_test.py b/test/core/rules/rule_test_cases_test.py
index 514ed9aed..2813ac3b6 100644
--- a/test/core/rules/rule_test_cases_test.py
+++ b/test/core/rules/rule_test_cases_test.py
@@ -1,31 +1,20 @@
 """Runs the rule test cases."""
 import os
-from glob import glob
-
 import pytest
-import oyaml as yaml
-
-from sqlfluff.core.config import FluffConfig
+from sqlfluff.testing.rules import (
+    load_test_cases,
+    rules__test_helper,
+    get_rule_from_set,
+)
 from sqlfluff.core.rules.doc_decorators import is_fix_compatible
-from .std_test import get_rule_from_set, rules__test_helper, RuleTestCase
-
-ids = []
-test_cases = []
+from sqlfluff.core.config import FluffConfig
 
-test_cases_path = os.path.join(
-    os.path.abspath(os.path.dirname(__file__)), "test_cases", "*.yml"
+ids, test_cases = load_test_cases(
+    test_cases_path=os.path.join(
+        os.path.abspath(os.path.dirname(__file__)), "test_cases", "*.yml"
+    )
 )
 
-for path in sorted(glob(test_cases_path)):
-    with open(path) as f:
-        raw = f.read()
-
-    y = yaml.safe_load(raw)
-
-    rule = y.pop("rule")
-    ids.extend([rule + "_" + t for t in y])
-    test_cases.extend([RuleTestCase(rule=rule, **v) for k, v in y.items()])
-
 
 @pytest.mark.parametrize("test_case", test_cases, ids=ids)
 def test__rule_test_case(test_case):
diff --git a/test/core/rules/std_test.py b/test/core/rules/std_test.py
index 45529e880..a240aaff2 100644
--- a/test/core/rules/std_test.py
+++ b/test/core/rules/std_test.py
@@ -1,113 +1,24 @@
 """Tests for the standard set of rules."""
-from typing import NamedTuple
-
 import pytest
 
 from sqlfluff.core import Linter
-from sqlfluff.core.errors import SQLParseError
 from sqlfluff.core.rules.base import BaseCrawler, LintResult, LintFix
-from sqlfluff.core.rules import std_rule_set
+from sqlfluff.core.rules import get_ruleset
 from sqlfluff.core.rules.doc_decorators import document_configuration
 from sqlfluff.core.config import FluffConfig
+from sqlfluff.testing.rules import (
+    assert_rule_raises_violations_in_file,
+    get_rule_from_set,
+)
 
 from test.fixtures.dbt.templater import (  # noqa
     DBT_FLUFF_CONFIG,
     in_dbt_project_dir,
     dbt_templater,
 )
-
-
-class RuleTestCase(NamedTuple):
-    """Used like a dataclass by rule tests."""
-
-    rule: str = None
-    desc: str = None
-    pass_str: str = None
-    fail_str: str = None
-    fix_str: str = None
-    configs: dict = None
-    skip: str = None
-
-
-def get_rule_from_set(code, config):
-    """Fetch a rule from the rule set."""
-    for r in std_rule_set.get_rulelist(config=config):
-        if r.code == code:
-            return r
-    raise ValueError("{0!r} not in {1!r}".format(code, std_rule_set))
-
-
-def assert_rule_fail_in_sql(code, sql, configs=None):
-    """Assert that a given rule does fail on the given sql."""
-    # Set up the config to only use the rule we are testing.
-    cfg = FluffConfig(configs=configs, overrides={"rules": code})
-    # Lint it using the current config (while in fix mode)
-    linted = Linter(config=cfg).lint_string(sql, fix=True)
-    lerrs = linted.get_violations()
-    print("Errors Found: {0}".format(lerrs))
-    parse_errors = list(filter(lambda v: type(v) == SQLParseError, lerrs))
-    if parse_errors:
-        pytest.fail(f"Found the following parse errors in test case: {parse_errors}")
-    if not any(v.rule.code == code for v in lerrs):
-        pytest.fail(
-            "No {0} failures found in query which should fail.".format(code),
-            pytrace=False,
-        )
-    # The query should already have been fixed if possible so just return the raw.
-    return linted.tree.raw
-
-
-def assert_rule_pass_in_sql(code, sql, configs=None):
-    """Assert that a given rule doesn't fail on the given sql."""
-    # Configs allows overrides if we want to use them.
-    cfg = FluffConfig(configs=configs)
-    r = get_rule_from_set(code, config=cfg)
-    parsed = Linter(config=cfg).parse_string(sql)
-    if parsed.violations:
-        pytest.fail(parsed.violations[0].desc() + "\n" + parsed.tree.stringify())
-    print("Parsed:\n {0}".format(parsed.tree.stringify()))
-    lerrs, _, _, _ = r.crawl(parsed.tree, dialect=cfg.get("dialect_obj"))
-    print("Errors Found: {0}".format(lerrs))
-    if any(v.rule.code == code for v in lerrs):
-        pytest.fail(
-            "Found {0} failures in query which should pass.".format(code), pytrace=False
-        )
-
-
-def assert_rule_raises_violations_in_file(rule, fpath, violations, fluff_config):
-    """Assert that a given rule raises given errors in specific positions of a file."""
-    lntr = Linter(config=fluff_config)
-    lnt = lntr.lint_path(fpath)
-    # Reformat the test data to match the format we're expecting. We use
-    # sets because we really don't care about order and if one is missing,
-    # we don't care about the orders of the correct ones.
-    assert set(lnt.check_tuples()) == {(rule, v[0], v[1]) for v in violations}
-
-
-def rules__test_helper(test_case):
-    """Test that a rule passes/fails on a set of test_cases.
-
-    Optionally, also test the fixed string if provided in the test case.
-    """
-    if test_case.skip:
-        pytest.skip(test_case.skip)
-
-    if test_case.pass_str:
-        assert_rule_pass_in_sql(
-            test_case.rule,
-            test_case.pass_str,
-            configs=test_case.configs,
-        )
-    if test_case.fail_str:
-        res = assert_rule_fail_in_sql(
-            test_case.rule,
-            test_case.fail_str,
-            configs=test_case.configs,
-        )
-        # If a `fixed` value is provided then check it matches
-        if test_case.fix_str:
-            assert res == test_case.fix_str
-        return res
+from test.fixtures.rules.L000 import Rule_L000
+from test.fixtures.rules.S000 import Rule_S000
+from sqlfluff.core.rules.std import get_rules_from_path
 
 
 class Rule_T042(BaseCrawler):
@@ -288,7 +199,7 @@ def test_improper_configs_are_rejected(rule_config_dict):
     """Ensure that unsupported configs raise a ValueError."""
     config = FluffConfig(configs={"rules": rule_config_dict})
     with pytest.raises(ValueError):
-        std_rule_set.get_rulelist(config)
+        get_ruleset().get_rulelist(config)
 
 
 def test_rules_cannot_be_instantiated_without_declared_configs():
@@ -328,14 +239,42 @@ def test_rules_configs_are_dynamically_documented():
 
 def test_rule_exception_is_caught_to_validation():
     """Assert that a rule that throws an exception on _eval returns it as a validation."""
+    std_rule_set = get_ruleset()
 
     @std_rule_set.register
-    class Rule_LXXX(BaseCrawler):
+    class Rule_T000(BaseCrawler):
         """Rule that throws an exception."""
 
         def _eval(self, segment, parent_stack, **kwargs):
             raise Exception("Catch me or I'll deny any linting results from you")
 
-    linter = Linter(config=FluffConfig(overrides=dict(rules="LXXX")))
+    linter = Linter(
+        config=FluffConfig(overrides=dict(rules="T000")),
+        user_rules=[Rule_T000],
+    )
+
+    assert linter.lint_string("select 1").check_tuples() == [("T000", 1, 1)]
+
+
+def test_std_rule_import_fail_bad_naming():
+    """Check that rule import from file works."""
+    assert get_rules_from_path(
+        rules_path="test/fixtures/rules/*.py", base_module="test.fixtures.rules"
+    ) == [Rule_L000, Rule_S000]
+
+    with pytest.raises(AttributeError) as e:
+        get_rules_from_path(
+            rules_path="test/fixtures/rules/bad_rule_name/*.py",
+            base_module="test.fixtures.rules.bad_rule_name",
+        )
+
+    e.match("Rule classes must be named in the format of")
+
+
+def test_rule_set_return_informative_error_when_rule_not_registered():
+    """Assert that a rule that throws an exception on _eval returns it as a validation."""
+    cfg = FluffConfig()
+    with pytest.raises(ValueError) as e:
+        get_rule_from_set("L000", config=cfg)
 
-    assert linter.lint_string("select 1").check_tuples() == [("LXXX", 1, 1)]
+    e.match("'L000' not in")
diff --git a/test/fixtures/rules/L000.py b/test/fixtures/rules/L000.py
new file mode 100644
index 000000000..3dc699d24
--- /dev/null
+++ b/test/fixtures/rules/L000.py
@@ -0,0 +1,7 @@
+"""Test std rule import."""
+
+
+class Rule_L000:
+    """Test std rule import."""
+
+    pass
diff --git a/test/fixtures/rules/S000.py b/test/fixtures/rules/S000.py
new file mode 100644
index 000000000..cc1e81e5c
--- /dev/null
+++ b/test/fixtures/rules/S000.py
@@ -0,0 +1,7 @@
+"""Test std rule import."""
+
+
+class Rule_S000:
+    """Test std rule import."""
+
+    pass
diff --git a/test/fixtures/rules/bad_rule_name/E000.py b/test/fixtures/rules/bad_rule_name/E000.py
new file mode 100644
index 000000000..243b23564
--- /dev/null
+++ b/test/fixtures/rules/bad_rule_name/E000.py
@@ -0,0 +1,7 @@
+"""Test std rule import."""
+
+
+class E000:
+    """This will fail to import because it does not start with Rule_."""
+
+    pass
diff --git a/test/test_testing.py b/test/test_testing.py
new file mode 100644
index 000000000..8e35232d3
--- /dev/null
+++ b/test/test_testing.py
@@ -0,0 +1,46 @@
+"""Test the sqlfluff.testing module."""
+
+from _pytest.outcomes import Failed, Skipped
+import pytest
+from sqlfluff.testing.rules import (
+    assert_rule_fail_in_sql,
+    assert_rule_pass_in_sql,
+    rules__test_helper,
+    RuleTestCase,
+)
+
+
+def test_assert_rule_fail_in_sql_handle_parse_error():
+    """Util assert_rule_fail_in_sql should handle parse errors."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_fail_in_sql(code="L000", sql="select from")
+    failed_test.match("Found the following parse errors in test case:")
+
+
+def test_assert_rule_fail_in_sql_should_fail_queries_that_unexpectedly_pass():
+    """Util assert_rule_fail_in_sql should fail tests when a query passes rules that it violates."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_fail_in_sql(code="L001", sql="select 1")
+    failed_test.match("No L001 failures found in query which should fail")
+
+
+def test_assert_rule_pass_in_sql_should_handle_parse_error():
+    """Util assert_rule_pass_in_sql should handle parse errors."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_pass_in_sql(code="L001", sql="select from")
+    failed_test.match("Found unparsable section:")
+
+
+def test_assert_rule_pass_in_sql_should_fail_when_there_are_violations():
+    """Util assert_rule_pass_in_sql should fail when there are violations."""
+    with pytest.raises(Failed) as failed_test:
+        assert_rule_pass_in_sql(code="L005", sql="select a , b from t")
+    failed_test.match("Found L005 failures in query which should pass")
+
+
+def test_rules__test_helper_skipped_when_test_case_skipped():
+    """Util rules__test_helper should skip the test when test case is "skipped"."""
+    rule_test_case = RuleTestCase(skip="Skip this one for now")
+    with pytest.raises(Skipped) as skipped_test:
+        rules__test_helper(rule_test_case)
+    skipped_test.match("Skip this one for now")