Repository: pre-commit/pre-commit

Issue:
Hooks pre-fetching
Hi,

I want to suggest introducing a command for downloading all required hooks forcefully. The use case is simple:
1. I pack everything needed for testing into container
2. I deploy that into CI
3. It gets built and tests are ran

I'd like to separate fetching of the hooks and pre-installing them into a build step. This would allow me use caching for containers, saving time for tests.

P.S. Perhaps it's needed to add some argument to `run` command, so that it won't try downloading stuff when executing tests.


Discussion:
We use `pre-commit install --install-hooks` to accomplish this for our jobs, does this work for you?
Thanks for the suggestion. It doesn't look installing all the dependencies into `$HOME/.pre-commit/` env, like `pre-commit run` does.
Can you provide some additional output?  The two commands run the same code:

## install --install-hooks
- https://github.com/pre-commit/pre-commit/blob/8837cfa7ffcc419216d4e01392cee0f1ceee9c88/pre_commit/commands/install_uninstall.py#L86

## run
- https://github.com/pre-commit/pre-commit/blob/8837cfa7ffcc419216d4e01392cee0f1ceee9c88/pre_commit/commands/run.py#L159
- https://github.com/pre-commit/pre-commit/blob/8837cfa7ffcc419216d4e01392cee0f1ceee9c88/pre_commit/runner.py#L41-L46
You are right, there was no output, because the hooks were installed before that.

Now I've faced another problem: whenever I run `pre-commit run --all-files` it tries upgrading them.

I need some way to prevent that. Any ideas?
It shouldn't, can you show some output or something reproducible?
ah I took a peek at your [.pre-commit-config.yaml](https://github.com/GDG-Ukraine/gdg.org.ua/blob/c7f5c91e326ec7933f936c926928808ddfe0fde7/.pre-commit-config.yaml) and it's probably because you're using unsupported `master` as `sha`. You can read more about why this doesn't quite work the way you want it to: https://github.com/pre-commit/pre-commit/issues/158#issuecomment-54103765

The suggested workflow for reproducibility is to use a sha or tag in that field (and periodically upgrade using `pre-commit autoupdate`).  Actually I think `pre-commit autoupdate` will fix your current situation automatically
Oh, I see.. Thanks for explanation.

My goal is to avoid having git installed in container during runtime. Normally I have it as a build dependency only. But it is running git for checks:
```
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at ~/.pre-commit/pre-commit.log
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/git.py", line 20, in get_root
    return cmd_output('git', 'rev-parse', '--show-toplevel')[1].strip()
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/util.py", line 188, in cmd_output
    returncode, cmd, retcode, output=(stdout, stderr),
pre_commit.util.CalledProcessError: Command: ('git', 'rev-parse', '--show-toplevel')
Return code: 1
Expected return code: 0
Output: 
    Executable `git` not found
Errors: (none)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/error_handler.py", line 47, in error_handler
    yield
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/main.py", line 166, in main
    runner = Runner.create(args.config)
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/runner.py", line 28, in create
    root = git.get_root()
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/git.py", line 23, in get_root
    'git failed. Is it installed, and are you in a Git repository '
pre_commit.errors.FatalError: git failed. Is it installed, and are you in a Git repository directory?
```

Any chance to turn the check off completely? Given that all hooks will already be downloaded and set up at build time.
Ah,  pre-commit uses git to both find the top level of the repository and acquire a file list to run hooks against.  The failure here is actually while pre-commit is detecting the top level 


In other words, `git` is an essential runtime dependency even when the hooks are already preinstalled.  I think this becomes wontfix?
Well, if it cannot be easily substituted with just a python code, then I think yes. You may close this issue.
Yeah it'd have to reimplement git internals which I'm not really interested in doing :)
@asottile It seems we need to re-open the question.
Project workdir in the container is just a folder of the repo at the host machine in RW mode. Thus when it changes the pre-commit hook of the repo it starts pointing to pre-commit inside of the container, which breaks the hook for the host machine.
Any ideas? Probably introducing some extra option would help.
This is an inherent problem with prefix installation.  If you're using shebangs of executables then the paths need to be the same inside and outside the container.  There's not really much that can be done to make them work at different paths since both the environment setup and installation are handled externally (through virtualenv,  setuptools, and pip). 

If possible,  I'd suggest mounting at the same path as you installed the hooks and it should work
Well, I just don't want it to change `.git/hooks/pre-commit` while running inside of container. Thus I would prefer some `--no-not-change-git-hook` CLI argument or so.
I'll reopen for now, mostly because I'm curious of your workflow.  Can you give me a comprehensive list of commands that you're running and whether they're inside / outside of the container as well as the (or an approximation of) the docker command you're running (mostly interested in the mounts).  With this I can hopefully suggest either an existing workflow which accomplishes what you want or perhaps a feature if I think it's necessary to do what you want.
For development I use [vagga](https://vagga.readthedocs.io/en/latest/) — containerization tool without daemons. Like docker, it also builds and runs LXC containers. But, unlike docker, it runs everything in userspace and isn't meant to be used in production (there's a supervisor [lithos](https://lithos.readthedocs.io/en/latest/), systemd-nspawn and lots of other tools for this).

The config is:
```yaml
containers:
  mysql:
    setup:
      - !Alpine v3.4
      - !Install
        - mariadb
        - mariadb-client
      - !EnsureDir /data
      - !EnsureDir /run
    environ:
      DB_DATABASE: gdg
      DB_USERNAME: mysql
      DB_PASSWORD: mysql
      DB_HOST: 127.0.0.1
      DB_PORT: 3307
      DB_DATA_DIR: /data
    volumes:
      /data: !Persistent
        name: mysql-data
        init-command: _init-mysql
      /run: !Tmpfs
        mode: 0o766
        subdirs:
          mysqld:

  app:
    setup:
      - !Ubuntu xenial
      - !UbuntuUniverse
      - &app-build-deps !BuildDeps
        - git
        - mercurial
        - python3.5-dev
      - !Install
        - ca-certificates
        - python3.5
      - !PipConfig
        dependencies: true
        python-exe: python3.5
      - !Depends setup.py
      - !Py3Requirements requirements/dev.txt
      - !NpmInstall [bower]
      - !Sh bower install --allow-root
      - !EnsureDir /mnt/db_host
    environ-file: /work/.env

  test:
    setup:
      - !Container app
      - !BuildDeps *app-build-deps
      - !Depends .pre-commit-config.yaml
      - !Py3Requirements requirements/test.txt
      - !Py3Requirements requirements/test-env.txt
      # Git is needed for pre-commit in runtime. Ref:
      # github.com/pre-commit/pre-commit/issues/456#issuecomment-269653630
      - !Install [git]
      # Shadow git-hooks dir, so that pre-commit won't break git hooks in host
      # Ref: github.com/pre-commit/pre-commit/issues/456#issuecomment-269856503
      - !Sh mount -t tmpfs none /work/.git/hooks
      - !Sh HOME=/root pre-commit install --install-hooks
    environ:
      # Ref:
      # github.com/pre-commit/pre-commit-hooks/pull/161#issuecomment-269662841
      LANG: en_US.UTF-8
      BLUEBERRYPY_CONFIG: "{}"
      NOSE_TESTCONFIG_AUTOLOAD_YAML: "config/test/app.yml"

commands:
  _init-mysql: !Command
    description: Initialize mysql database
    container: mysql
    run: |
      mysql_install_db --datadir=$DB_DATA_DIR
      mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                  --bind-address=$DB_HOST --port=$DB_PORT \
                  --no-auto-restart --no-watch
      while [ ! -S /run/mysqld/mysqld.sock ]; do
        sleep .2
      done  # wait for server to be ready
      mysqladmin create $DB_DATABASE
      mysql -e "CREATE USER '$DB_USERNAME'@'localhost' IDENTIFIED BY '$DB_PASSWORD';"
      mysql -e "GRANT ALL PRIVILEGES ON $DB_DATABASE.* TO '$DB_USERNAME'@'localhost';"
      mysql -e "FLUSH PRIVILEGES;"
  clean-db: !Command
    description: Cleanup mysql database
    container: mysql
    run: |
      mysql_install_db --datadir=$DB_DATA_DIR
      mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                  --bind-address=$DB_HOST --port=$DB_PORT \
                  --no-auto-restart --no-watch
      while [ ! -S /run/mysqld/mysqld.sock ]; do
        sleep .2
      done  # wait for server to be ready
      mysqladmin -f drop "$DB_DATABASE"
      mysqladmin create "$DB_DATABASE"
      mysql -e "GRANT ALL PRIVILEGES ON $DB_DATABASE.* TO '$DB_USERNAME'@'localhost';"
      mysql -e "FLUSH PRIVILEGES;"
  blueberrypy: !Command
    description: |
      Run blueberrypy command (you have to provide command and arguments by yourself)
    container: app
    run: [ blueberrypy ]

  mysql: !Command
    description: Run RDBMS shell
    container: mysql
    run: |
      mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                  --bind-address=$DB_HOST --port=$DB_PORT \
                  --no-auto-restart --no-watch
      while [ ! -S /run/mysqld/mysqld.sock ]; do
        sleep .2
      done
      mysql -D $DB_DATABASE
  run: !Supervise
    description: Run application in development mode
    mode: stop-on-failure
    children:
      run-app: !Command
        container: app
        run: |
          touch /work/.dbcreation  # Create lock file
          while  [ -f /work/.dbcreation ]; do  # Acquire lock
            sleep .2
          done
          current_version=$(alembic -c config/alembic.ini -x environment=dev current)
          head_version=$(alembic -c config/alembic.ini -x environment=dev heads)
          if [ "${current_version}" != "${head_version}" ]; then
            alembic -c config/alembic.ini -x environment=dev upgrade head
          fi
          if [ -z "${current_version}" ]; then
            load_gdg_fixtures "$DATABASE_URL" src/GDGUkraine/fixtures/fixtures.yaml || exit 1
          fi
          blueberrypy serve -b 0.0.0.0:8080
      run-db: !Command
        container: mysql
        run: |
          mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                      --bind-address=$DB_HOST --port=$DB_PORT \
                      --no-auto-restart --no-watch
          while [ ! -S /run/mysqld/mysqld.sock ]; do
            sleep .2
          done  # wait for server to be ready
          rm -f /work/.dbcreation  # Release lock
          while :; do  # Emulate infinite loop
            sleep 1d;
          done
  lint: !Command
    description: Run linters for gdg.org.ua project
    container: test
    run: pre-commit run --all-files

  'py.test': !Command
    description: Run tests for gdg.org.ua project
    container: test
    run: [py.test, --cov, -v]

  test: !Command
    description: Run tests for gdg.org.ua project
    container: test
    run: py.test --cov -v src/tests/
```

So when I run `vagga lint` it:
1) builds `app` and `test` containers *if they don't exist or any of their dependencies in project changed (`!Depend` or `requirements*.txt` etc.)*
1.0) during the build my current directory (with config, which is a repo dir) is being mounted as `/work`, rw
1.1) at this stage all needed dependencies are being installed
1.2) during the build I hacked around the issue and shadowed `/work/.git/hooks` with `tmpfs`, but it doesn't look like a right thing to do
1.3) `HOME=/root pre-commit install --install-hooks` installs hooks to dir, which is `$HOME` at runtime
2) executes `pre-commit run --all-files` command inside of `test` container
2.0) during the run my project dir is also mounted as `/work`, rw

I'm doing `git commit`, which runs pre-commit outside of container, that is why I don't want the build process to influence any git configuration at host.
ok.  In that case, I'll add a `pre-commit install-hooks` cmdline, I think that's probably the cleanest way to do this

Resolving patch:
diff --git a/pre_commit/commands/install_uninstall.py b/pre_commit/commands/install_uninstall.py
--- a/pre_commit/commands/install_uninstall.py
+++ b/pre_commit/commands/install_uninstall.py
@@ -82,12 +82,16 @@ def install(runner, overwrite=False, hooks=False, hook_type='pre-commit'):
 
     # If they requested we install all of the hooks, do so.
     if hooks:
-        for repository in runner.repositories:
-            repository.require_installed()
+        install_hooks(runner)
 
     return 0
 
 
+def install_hooks(runner):
+    for repository in runner.repositories:
+        repository.require_installed()
+
+
 def uninstall(runner, hook_type='pre-commit'):
     """Uninstall the pre-commit hooks."""
     hook_path = runner.get_hook_path(hook_type)
diff --git a/pre_commit/main.py b/pre_commit/main.py
--- a/pre_commit/main.py
+++ b/pre_commit/main.py
@@ -12,6 +12,7 @@
 from pre_commit.commands.autoupdate import autoupdate
 from pre_commit.commands.clean import clean
 from pre_commit.commands.install_uninstall import install
+from pre_commit.commands.install_uninstall import install_hooks
 from pre_commit.commands.install_uninstall import uninstall
 from pre_commit.commands.run import run
 from pre_commit.error_handler import error_handler
@@ -78,6 +79,17 @@ def main(argv=None):
         default='pre-commit',
     )
 
+    install_hooks_parser = subparsers.add_parser(
+        'install-hooks',
+        help=(
+            'Install hook environemnts for all environemnts in the config '
+            'file.  You may find `pre-commit install --install-hooks` more '
+            'useful.'
+        ),
+    )
+    _add_color_option(install_hooks_parser)
+    _add_config_option(install_hooks_parser)
+
     uninstall_parser = subparsers.add_parser(
         'uninstall', help='Uninstall the pre-commit script.',
     )
@@ -171,6 +183,8 @@ def main(argv=None):
                 runner, overwrite=args.overwrite, hooks=args.install_hooks,
                 hook_type=args.hook_type,
             )
+        elif args.command == 'install-hooks':
+            return install_hooks(runner)
         elif args.command == 'uninstall':
             return uninstall(runner, hook_type=args.hook_type)
         elif args.command == 'clean':


Test patch:
diff --git a/tests/commands/install_uninstall_test.py b/tests/commands/install_uninstall_test.py
--- a/tests/commands/install_uninstall_test.py
+++ b/tests/commands/install_uninstall_test.py
@@ -13,6 +13,7 @@
 import pre_commit.constants as C
 from pre_commit.commands.install_uninstall import IDENTIFYING_HASH
 from pre_commit.commands.install_uninstall import install
+from pre_commit.commands.install_uninstall import install_hooks
 from pre_commit.commands.install_uninstall import is_our_pre_commit
 from pre_commit.commands.install_uninstall import is_previous_pre_commit
 from pre_commit.commands.install_uninstall import PREVIOUS_IDENTIFYING_HASHES
@@ -460,6 +461,20 @@ def test_installs_hooks_with_hooks_True(
         assert PRE_INSTALLED.match(output)
 
 
+def test_install_hooks_command(tempdir_factory, mock_out_store_directory):
+    path = make_consuming_repo(tempdir_factory, 'script_hooks_repo')
+    with cwd(path):
+        runner = Runner(path, C.CONFIG_FILE)
+        install(runner)
+        install_hooks(runner)
+        ret, output = _get_commit_output(
+            tempdir_factory, pre_commit_home=mock_out_store_directory,
+        )
+
+        assert ret == 0
+        assert PRE_INSTALLED.match(output)
+
+
 def test_installed_from_venv(tempdir_factory):
     path = make_consuming_repo(tempdir_factory, 'script_hooks_repo')
     with cwd(path):
diff --git a/tests/main_test.py b/tests/main_test.py
--- a/tests/main_test.py
+++ b/tests/main_test.py
@@ -15,17 +15,19 @@
 @pytest.yield_fixture
 def mock_commands():
     with mock.patch.object(main, 'autoupdate') as autoupdate_mock:
-        with mock.patch.object(main, 'clean') as clean_mock:
+        with mock.patch.object(main, 'install_hooks') as install_hooks_mock:
             with mock.patch.object(main, 'install') as install_mock:
                 with mock.patch.object(main, 'uninstall') as uninstall_mock:
                     with mock.patch.object(main, 'run') as run_mock:
-                        yield auto_namedtuple(
-                            autoupdate_mock=autoupdate_mock,
-                            clean_mock=clean_mock,
-                            install_mock=install_mock,
-                            uninstall_mock=uninstall_mock,
-                            run_mock=run_mock,
-                        )
+                        with mock.patch.object(main, 'clean') as clean_mock:
+                            yield auto_namedtuple(
+                                autoupdate_mock=autoupdate_mock,
+                                clean_mock=clean_mock,
+                                install_mock=install_mock,
+                                install_hooks_mock=install_hooks_mock,
+                                uninstall_mock=uninstall_mock,
+                                run_mock=run_mock,
+                            )
 
 
 class CalledExit(Exception):
@@ -121,6 +123,12 @@ def test_run_command(mock_commands):
     assert_only_one_mock_called(mock_commands)
 
 
+def test_install_hooks_command(mock_commands):
+    main.main(('install-hooks',))
+    assert mock_commands.install_hooks_mock.call_count == 1
+    assert_only_one_mock_called(mock_commands)
+
+
 def test_no_commands_run_command(mock_commands):
     main.main([])
     assert mock_commands.run_mock.call_count == 1