Repository: pre-commit/pre-commit

Issue:
Add support for pcre / scripts / system hooks definition in .pre-commit-config.yaml
Everything is in the title :)

_Rationale:_ a `pre-commit` user shouldn't have to setup a git repository to configure a pre-commit check that can be defined in 5 lines or less.

_Example:_ taken the from [do_not_commit](https://github.com/pricematch/pricematch-pre-commit-hooks/blob/master/hooks.yaml) hook:

```
- id: do_not_commit
  name: Block if "DO NOT COMMIT" is found
  entry: DO NOT COMMIT
  language: pcre
  files: ^(.*)$
```

_Suggested solutions:_
1. Allow for pcre / scripts / system hooks definition in _.pre-commit-config.yaml_
2. Allow for the `repo` field in _.pre-commit-config.yaml_ to point to a subfolder (of the git repo configured with `pre-commit`) that contains a _hooks.yaml_ .
   This currently crashes because `pre-commit` expect to find git repository in the folder point by `repo`.



Discussion:
I think this is along similar lines of this: https://github.com/pre-commit/pre-commit/issues/173

And is kind-of possible: https://github.com/Yelp/venv-update/blob/a5960acab7101a1e70c57945b2038fef9d005aed/.pre-commit-config.yaml#L15-L22

I'd certainly like to make this easier though :)

Yes ! The `Yelp/venv-update` example is exactly what I wanted !
If you support this, maybe mention it in the docs ?
Although maybe allowing a `repo: local` value would be cleaner than having to point it to an existing dummy github repo.

Yeah that way _works_ currently, I wouldn't call it supported yet since it's kinda clunky and I want a more straightforward way of doing that.  That method is unlikely to break though.  I imagine the actual implementation will be similar

Ok. Then you can resolve this ticket if you want to track this feature elsewhere.
And thank you for your detailed answers !

I went on and hacked around it: what do you think of the solution described in #223 ?

I'm going to clarify here since I think I'm a bit confused :)

I'd like to make @bukzor's config work when it changes from:

``` yaml
-   repo: git://github.com/bukzor/pre-commit-system-hook.git
    sha: 64a3e3f0d11f74ccd0498ea3149c8e177dd9989c
    hooks:
    -   id: system
        name: PyLint
        entry: python -m pylint.__main__
        language: system
        files: \.py$
```

to 

``` yaml
-   repo: pre-commit-locally-configured
    hooks:
    -   id: pylint
        name: PyLint
        entry: python -m pylint.__main__
        language: system
        files: \.py$
```

I think (but I might be a bit off!) that all we would need to change to support this is:
- Change the config schema to allow for missing sha in the case the repo is our special value
- In the case that our repo is the special value, the repository is a special list which contains "all hooks" (that is to say that anything we configure in our config will match a hook in this special repo)
- Allow the current code which merges the config dict on top of the manifest dict to continue to work (might need it in another place as well)

Does this sound right? (mostly wanted to just get a checklist together so it's clear what the plan is here :D)

Sounds alright, + adding some tests :)
I won't have time to craft the commit today, but I plan to submit the pull request over the week-end.

Repo: null makes sense to me.
Just don't validate the sha for language: system.

Also: can we put system-style hooks at the top level? It doesn't actually
make sense to nest then under a repo at all.

Just ideas. If it makes things a mess, disregard.

--phone is hard.
On May 7, 2015 6:17 PM, "Anthony Sottile" notifications@github.com wrote:

> I'm going to clarify here since I think I'm a bit confused :)
> 
> I'd like to make @bukzor https://github.com/bukzor's config work when
> it changes from:
> -   repo: git://github.com/bukzor/pre-commit-system-hook.git
>   sha: 64a3e3f0d11f74ccd0498ea3149c8e177dd9989c
>   hooks:
>   -   id: system
>     name: PyLint
>     entry: python -m pylint.**main**
>     language: system
>     files: .py$
> 
> to
> -   repo: pre-commit-locally-configured
>   hooks:
>   -   id: pylint
>     name: PyLint
>     entry: python -m pylint.**main**
>     language: system
>     files: .py$
> 
> I think (but I might be a bit off!) that all we would need to change to
> support this is:
> - Change the config schema to allow for missing sha in the case the
>   repo is our special value
> - In the case that our repo is the special value, the repository is a
>   special list which contains "all hooks" (that is to say that anything we
>   configure in our config will match a hook in this special repo)
> - Allow the current code which merges the config dict on top of the
>   manifest dict to continue to work (might need it in another place as well)
> 
> Does this sound right? (mostly wanted to just get a checklist together so
> it's clear what the plan is here :D)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pre-commit/pre-commit/issues/219#issuecomment-100060347
> .

> Repo: null makes sense to me.

Sounds nice & simple, but null / None is often a design smell: it is often better to use an "Optional" construct, a NullObject or the SpecialCase pattern. I'll stick to the magic string constant.

> can we put system-style hooks at the top level?

I don't understand what you meant here. I complete agree it doesn't make sense to put them in a repo, but why put them at the YAML config file hierarchy root ? I don't quite see what change to the plan @asottile detailed you are suggesting there. Could you further explain ?

> Also: can we put system-style hooks at the top level?

@bukzor this would entail a breaking change to the config file, I don't really want to do that at this time.

EDIT: null also seems fine to me as well

Is fine. It probably doesn't make sense for the list to be heterogeneous.

--phone is hard.
On May 8, 2015 9:54 AM, "Anthony Sottile" notifications@github.com wrote:

> Also: can we put system-style hooks at the top level?
> @bukzor https://github.com/bukzor this would entail a breaking change
> to the config file, I don't really want to do that at this time.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pre-commit/pre-commit/issues/219#issuecomment-100297345
> .


Resolving patch:
diff --git a/pre_commit/clientlib/validate_config.py b/pre_commit/clientlib/validate_config.py
--- a/pre_commit/clientlib/validate_config.py
+++ b/pre_commit/clientlib/validate_config.py
@@ -6,6 +6,13 @@
 from pre_commit.errors import FatalError
 
 
+_LOCAL_HOOKS_MAGIC_REPO_STRING = 'local'
+
+
+def is_local_hooks(repo_entry):
+    return repo_entry['repo'] == _LOCAL_HOOKS_MAGIC_REPO_STRING
+
+
 class InvalidConfigError(FatalError):
     pass
 
@@ -53,7 +60,12 @@ def try_regex(repo, hook, value, field_name):
 
 def validate_config_extra(config):
     for repo in config:
-        if 'sha' not in repo:
+        if is_local_hooks(repo):
+            if 'sha' in repo:
+                raise InvalidConfigError(
+                    '"sha" property provided for local hooks'
+                )
+        elif 'sha' not in repo:
             raise InvalidConfigError(
                 'Missing "sha" field for repository {0}'.format(repo['repo'])
             )
diff --git a/pre_commit/commands/run.py b/pre_commit/commands/run.py
--- a/pre_commit/commands/run.py
+++ b/pre_commit/commands/run.py
@@ -18,15 +18,6 @@
 logger = logging.getLogger('pre_commit')
 
 
-class HookExecutor(object):
-    def __init__(self, hook, invoker):
-        self.hook = hook
-        self._invoker = invoker
-
-    def invoke(self, filenames):
-        return self._invoker(self.hook, filenames)
-
-
 def _get_skips(environ):
     skips = environ.get('SKIP', '')
     return set(skip.strip() for skip in skips.split(',') if skip.strip())
@@ -80,8 +71,7 @@ def get_filenames(args, include_expr, exclude_expr):
     return getter(include_expr, exclude_expr)
 
 
-def _run_single_hook(hook_executor, args, write, skips=frozenset()):
-    hook = hook_executor.hook
+def _run_single_hook(hook, repo, args, write, skips=frozenset()):
     filenames = get_filenames(args, hook['files'], hook['exclude'])
     if hook['id'] in skips:
         _print_user_skipped(hook, write, args)
@@ -95,7 +85,7 @@ def _run_single_hook(hook_executor, args, write, skips=frozenset()):
     write(get_hook_message(_hook_msg_start(hook, args.verbose), end_len=6))
     sys.stdout.flush()
 
-    retcode, stdout, stderr = hook_executor.invoke(filenames)
+    retcode, stdout, stderr = repo.run_hook(hook, filenames)
 
     if retcode != hook['expected_return_value']:
         retcode = 1
@@ -119,19 +109,19 @@ def _run_single_hook(hook_executor, args, write, skips=frozenset()):
     return retcode
 
 
-def _run_hooks(hook_executors, args, write, environ):
+def _run_hooks(repo_hooks, args, write, environ):
     """Actually run the hooks."""
     skips = _get_skips(environ)
     retval = 0
-    for hook_executor in hook_executors:
-        retval |= _run_single_hook(hook_executor, args, write, skips)
+    for repo, hook in repo_hooks:
+        retval |= _run_single_hook(hook, repo, args, write, skips)
     return retval
 
 
-def get_hook_executors(runner):
+def get_repo_hooks(runner):
     for repo in runner.repositories:
-        for _, repo_hook in repo.hooks:
-            yield HookExecutor(repo_hook, repo.run_hook)
+        for _, hook in repo.hooks:
+            yield (repo, hook)
 
 
 def _has_unmerged_paths(runner):
@@ -159,13 +149,13 @@ def run(runner, args, write=sys_stdout_write_wrapper, environ=os.environ):
         ctx = staged_files_only(runner.cmd_runner)
 
     with ctx:
-        hook_executors = list(get_hook_executors(runner))
+        repo_hooks = list(get_repo_hooks(runner))
         if args.hook:
-            hook_executors = [
-                he for he in hook_executors
-                if he.hook['id'] == args.hook
+            repo_hooks = [
+                (repo, hook) for repo, hook in repo_hooks
+                if hook['id'] == args.hook
             ]
-            if not hook_executors:
+            if not repo_hooks:
                 write('No hook with id `{0}`\n'.format(args.hook))
                 return 1
-        return _run_hooks(hook_executors, args, write, environ)
+        return _run_hooks(repo_hooks, args, write, environ)
diff --git a/pre_commit/repository.py b/pre_commit/repository.py
--- a/pre_commit/repository.py
+++ b/pre_commit/repository.py
@@ -5,6 +5,10 @@
 
 from cached_property import cached_property
 
+from pre_commit import git
+from pre_commit.clientlib.validate_config import is_local_hooks
+from pre_commit.clientlib.validate_manifest import MANIFEST_JSON_SCHEMA
+from pre_commit.jsonschema_extensions import apply_defaults
 from pre_commit.languages.all import languages
 from pre_commit.manifest import Manifest
 from pre_commit.prefixed_command_runner import PrefixedCommandRunner
@@ -21,10 +25,13 @@ def __init__(self, repo_config, repo_path_getter):
 
     @classmethod
     def create(cls, config, store):
-        repo_path_getter = store.get_repo_path_getter(
-            config['repo'], config['sha']
-        )
-        return cls(config, repo_path_getter)
+        if is_local_hooks(config):
+            return LocalRepository(config)
+        else:
+            repo_path_getter = store.get_repo_path_getter(
+                config['repo'], config['sha']
+            )
+            return cls(config, repo_path_getter)
 
     @cached_property
     def repo_url(self):
@@ -111,3 +118,28 @@ def run_hook(self, hook, file_args):
         return languages[hook['language']].run_hook(
             self.cmd_runner, hook, file_args,
         )
+
+
+class LocalRepository(Repository):
+    def __init__(self, repo_config, repo_path_getter=None):
+        repo_path_getter = None
+        super(LocalRepository, self).__init__(repo_config, repo_path_getter)
+
+    @cached_property
+    def hooks(self):
+        return tuple(
+            (hook['id'], apply_defaults(hook, MANIFEST_JSON_SCHEMA['items']))
+            for hook in self.repo_config['hooks']
+        )
+
+    @cached_property
+    def cmd_runner(self):
+        return PrefixedCommandRunner(git.get_root())
+
+    @cached_property
+    def sha(self):
+        raise NotImplementedError
+
+    @cached_property
+    def manifest(self):
+        raise NotImplementedError


Test patch:
diff --git a/testing/fixtures.py b/testing/fixtures.py
--- a/testing/fixtures.py
+++ b/testing/fixtures.py
@@ -60,12 +60,16 @@ def write_config(directory, config):
         config_file.write(ordered_dump([config], **C.YAML_DUMP_KWARGS))
 
 
-def make_consuming_repo(tmpdir_factory, repo_source):
-    path = make_repo(tmpdir_factory, repo_source)
-    config = make_config_from_repo(path)
-    git_path = git_dir(tmpdir_factory)
+def add_config_to_repo(git_path, config):
     write_config(git_path, config)
     with cwd(git_path):
         cmd_output('git', 'add', C.CONFIG_FILE)
         cmd_output('git', 'commit', '-m', 'Add hooks config')
     return git_path
+
+
+def make_consuming_repo(tmpdir_factory, repo_source):
+    path = make_repo(tmpdir_factory, repo_source)
+    config = make_config_from_repo(path)
+    git_path = git_dir(tmpdir_factory)
+    return add_config_to_repo(git_path, config)
diff --git a/tests/clientlib/validate_config_test.py b/tests/clientlib/validate_config_test.py
--- a/tests/clientlib/validate_config_test.py
+++ b/tests/clientlib/validate_config_test.py
@@ -1,5 +1,6 @@
 from __future__ import unicode_literals
 
+import jsonschema
 import pytest
 
 from pre_commit.clientlib.validate_config import CONFIG_JSON_SCHEMA
@@ -25,7 +26,7 @@ def test_run(input, expected_output):
     assert run(input) == expected_output
 
 
-@pytest.mark.parametrize(('manifest_obj', 'expected'), (
+@pytest.mark.parametrize(('config_obj', 'expected'), (
     ([], False),
     (
         [{
@@ -66,8 +67,8 @@ def test_run(input, expected_output):
         False,
     ),
 ))
-def test_is_valid_according_to_schema(manifest_obj, expected):
-    ret = is_valid_according_to_schema(manifest_obj, CONFIG_JSON_SCHEMA)
+def test_is_valid_according_to_schema(config_obj, expected):
+    ret = is_valid_according_to_schema(config_obj, CONFIG_JSON_SCHEMA)
     assert ret is expected
 
 
@@ -121,3 +122,55 @@ def test_config_with_ok_exclude_regex_passes():
         CONFIG_JSON_SCHEMA,
     )
     validate_config_extra(config)
+
+
+@pytest.mark.parametrize('config_obj', (
+    [{
+        'repo': 'local',
+        'sha': 'foo',
+        'hooks': [{
+            'id': 'do_not_commit',
+            'name': 'Block if "DO NOT COMMIT" is found',
+            'entry': 'DO NOT COMMIT',
+            'language': 'pcre',
+            'files': '^(.*)$',
+        }],
+    }],
+))
+def test_config_with_local_hooks_definition_fails(config_obj):
+    with pytest.raises((
+        jsonschema.exceptions.ValidationError, InvalidConfigError
+    )):
+        jsonschema.validate(config_obj, CONFIG_JSON_SCHEMA)
+        config = apply_defaults(config_obj, CONFIG_JSON_SCHEMA)
+        validate_config_extra(config)
+
+
+@pytest.mark.parametrize('config_obj', (
+    [{
+        'repo': 'local',
+        'hooks': [{
+            'id': 'arg-per-line',
+            'name': 'Args per line hook',
+            'entry': 'bin/hook.sh',
+            'language': 'script',
+            'files': '',
+            'args': ['hello', 'world'],
+        }],
+    }],
+    [{
+        'repo': 'local',
+        'hooks': [{
+            'id': 'arg-per-line',
+            'name': 'Args per line hook',
+            'entry': 'bin/hook.sh',
+            'language': 'script',
+            'files': '',
+            'args': ['hello', 'world'],
+        }]
+    }],
+))
+def test_config_with_local_hooks_definition_passes(config_obj):
+    jsonschema.validate(config_obj, CONFIG_JSON_SCHEMA)
+    config = apply_defaults(config_obj, CONFIG_JSON_SCHEMA)
+    validate_config_extra(config)
diff --git a/tests/commands/run_test.py b/tests/commands/run_test.py
--- a/tests/commands/run_test.py
+++ b/tests/commands/run_test.py
@@ -14,10 +14,12 @@
 from pre_commit.commands.run import _has_unmerged_paths
 from pre_commit.commands.run import get_changed_files
 from pre_commit.commands.run import run
+from pre_commit.ordereddict import OrderedDict
 from pre_commit.runner import Runner
 from pre_commit.util import cmd_output
 from pre_commit.util import cwd
 from testing.auto_namedtuple import auto_namedtuple
+from testing.fixtures import add_config_to_repo
 from testing.fixtures import make_consuming_repo
 
 
@@ -81,7 +83,7 @@ def _test_run(repo, options, expected_outputs, expected_ret, stage):
         stage_a_file()
     args = _get_opts(**options)
     ret, printed = _do_run(repo, args)
-    assert ret == expected_ret
+    assert ret == expected_ret, (ret, expected_ret, printed)
     for expected_output_part in expected_outputs:
         assert expected_output_part in printed
 
@@ -313,9 +315,7 @@ def test_lots_of_files(mock_out_store_directory, tmpdir_factory):
     git_path = make_consuming_repo(tmpdir_factory, 'python_hooks_repo')
     with cwd(git_path):
         # Override files so we run against them
-        with io.open(
-            '.pre-commit-config.yaml', 'a+',
-        ) as config_file:
+        with io.open('.pre-commit-config.yaml', 'a+') as config_file:
             config_file.write('        files: ""\n')
 
         # Write a crap ton of files
@@ -334,3 +334,66 @@ def test_lots_of_files(mock_out_store_directory, tmpdir_factory):
             stderr=subprocess.STDOUT,
             env=env,
         )
+
+
+def test_local_hook_passes(
+        repo_with_passing_hook, mock_out_store_directory,
+):
+    config = OrderedDict((
+        ('repo', 'local'),
+        ('hooks', (OrderedDict((
+            ('id', 'pylint'),
+            ('name', 'PyLint'),
+            ('entry', 'python -m pylint.__main__'),
+            ('language', 'system'),
+            ('files', r'\.py$'),
+        )), OrderedDict((
+            ('id', 'do_not_commit'),
+            ('name', 'Block if "DO NOT COMMIT" is found'),
+            ('entry', 'DO NOT COMMIT'),
+            ('language', 'pcre'),
+            ('files', '^(.*)$'),
+        ))))
+    ))
+    add_config_to_repo(repo_with_passing_hook, config)
+
+    with io.open('dummy.py', 'w') as staged_file:
+        staged_file.write('"""TODO: something"""\n')
+    cmd_output('git', 'add', 'dummy.py')
+
+    _test_run(
+        repo_with_passing_hook,
+        options={},
+        expected_outputs=[''],
+        expected_ret=0,
+        stage=False
+    )
+
+
+def test_local_hook_fails(
+        repo_with_passing_hook, mock_out_store_directory,
+):
+    config = OrderedDict((
+        ('repo', 'local'),
+        ('hooks', [OrderedDict((
+            ('id', 'no-todo'),
+            ('name', 'No TODO'),
+            ('entry', 'grep -iI todo'),
+            ('expected_return_value', 1),
+            ('language', 'system'),
+            ('files', ''),
+        ))])
+    ))
+    add_config_to_repo(repo_with_passing_hook, config)
+
+    with io.open('dummy.py', 'w') as staged_file:
+        staged_file.write('"""TODO: something"""\n')
+    cmd_output('git', 'add', 'dummy.py')
+
+    _test_run(
+        repo_with_passing_hook,
+        options={},
+        expected_outputs=[''],
+        expected_ret=1,
+        stage=False
+    )
diff --git a/tests/repository_test.py b/tests/repository_test.py
--- a/tests/repository_test.py
+++ b/tests/repository_test.py
@@ -12,6 +12,7 @@
 from pre_commit.clientlib.validate_config import validate_config_extra
 from pre_commit.jsonschema_extensions import apply_defaults
 from pre_commit.languages.python import PythonEnv
+from pre_commit.ordereddict import OrderedDict
 from pre_commit.repository import Repository
 from pre_commit.util import cmd_output
 from pre_commit.util import cwd
@@ -377,3 +378,22 @@ def test_tags_on_repositories(in_tmpdir, tmpdir_factory, store):
     ret = repo_2.run_hook(repo_2.hooks[0][1], ['bar'])
     assert ret[0] == 0
     assert ret[1] == 'bar\nHello World\n'
+
+
+def test_local_repository():
+    config = OrderedDict((
+        ('repo', 'local'),
+        ('hooks', [OrderedDict((
+            ('id', 'do_not_commit'),
+            ('name', 'Block if "DO NOT COMMIT" is found'),
+            ('entry', 'DO NOT COMMIT'),
+            ('language', 'pcre'),
+            ('files', '^(.*)$'),
+        ))])
+    ))
+    local_repo = Repository.create(config, 'dummy')
+    with pytest.raises(NotImplementedError):
+        local_repo.sha
+    with pytest.raises(NotImplementedError):
+        local_repo.manifest
+    assert len(local_repo.hooks) == 1
diff --git a/tests/runner_test.py b/tests/runner_test.py
--- a/tests/runner_test.py
+++ b/tests/runner_test.py
@@ -5,8 +5,10 @@
 import os.path
 
 import pre_commit.constants as C
+from pre_commit.ordereddict import OrderedDict
 from pre_commit.runner import Runner
 from pre_commit.util import cwd
+from testing.fixtures import add_config_to_repo
 from testing.fixtures import git_dir
 from testing.fixtures import make_consuming_repo
 
@@ -52,6 +54,31 @@ def test_repositories(tmpdir_factory, mock_out_store_directory):
     assert len(runner.repositories) == 1
 
 
+def test_local_hooks(tmpdir_factory, mock_out_store_directory):
+    config = OrderedDict((
+        ('repo', 'local'),
+        ('hooks', (OrderedDict((
+            ('id', 'arg-per-line'),
+            ('name', 'Args per line hook'),
+            ('entry', 'bin/hook.sh'),
+            ('language', 'script'),
+            ('files', ''),
+            ('args', ['hello', 'world']),
+        )), OrderedDict((
+            ('id', 'do_not_commit'),
+            ('name', 'Block if "DO NOT COMMIT" is found'),
+            ('entry', 'DO NOT COMMIT'),
+            ('language', 'pcre'),
+            ('files', '^(.*)$'),
+        ))))
+    ))
+    git_path = git_dir(tmpdir_factory)
+    add_config_to_repo(git_path, config)
+    runner = Runner(git_path)
+    assert len(runner.repositories) == 1
+    assert len(runner.repositories[0].hooks) == 2
+
+
 def test_pre_commit_path():
     runner = Runner(os.path.join('foo', 'bar'))
     expected_path = os.path.join('foo', 'bar', '.git', 'hooks', 'pre-commit')