Repository: pre-commit/pre-commit

Issue:
Improve platform command line length limits
I have a question related to #510 

When I run `pre-commit run` all hooks work as expected (processing just changed files), but `pre-commit run -a` divides a list of files into 5 bulks and run the same hook in parallel executions. This creates unnecessary checks of files which belong to the same directory.

Is it possible to control that certain hooks should not be executed in parallel but rather pass all filenames to it?

Alternatively, is there an environment variable or something like that which says that pre-commit is running a hook for all files?


Discussion:
So pre-commit doesn't _yet_ run anything in parallel (as proposed in #510 -- and there will be a way to opt out of it once implemented).

What you're probably seeing is the batching of calls.  Passing all arguments to commands will not work in in all situations (there is a command length limit!).  On unixlike systems, coreutils provides a utility [`xargs`](http://man7.org/linux/man-pages/man1/xargs.1.html) to compute how many arguments to pass to avoid this limit.   pre-commit implements a similar helper but is [extremely timid](https://github.com/pre-commit/pre-commit/blob/51ac0e8bc0a1b61939283254c4bd5189c919f919/pre_commit/xargs.py#L8-L10) due to limits found on windows.

This limit can probably be increased on platforms where the limit is higher (if this can be detected?) and then you'll be much more likely to get one in a single batch.  I'd love a PR that does this :D
@antonbabenko no hooks operate in parallel yet; that issue is requesting the functionality, but it hasn't been implemented yet.

It's true that pre-commit may split files across multiple hook invocations if there are too many files, but this happens in serial and is sort of unavoidable (since there are limits on the length of `argv`, see `xargs --show-limits`; though pre-commit is a lot more conservative than what the actual max probably is on your system):

https://github.com/pre-commit/pre-commit/blob/51ac0e8bc0a1b61939283254c4bd5189c919f919/pre_commit/xargs.py#L8-L10
Thanks for the explanation. I am fine with the little overhead when running the same hook over the same directory.
The code hints here are:

- [`os.sysconf`](https://docs.python.org/3/library/os.html#os.sysconf)
- [`SC_ARG_MAX`](http://ftp.gnu.org/old-gnu/Manuals/glibc-2.2.3/html_chapter/libc_31.html#SEC648)
- [`xargs` implementation](http://git.savannah.gnu.org/cgit/findutils.git/tree/xargs/xargs.c)
    - `xargs` leaves room for posix mandated `2048` bytes, environ, and command.  It maxes by default at 128KiB (`131072` bytes).

And that none of this can be trusted on windows heh (so probably only do this if `sys.platform != 'win32'` (should also double check how `cygwin` responds here).
To clarify above, "by default" means "unless you specify `-s max-chars`".
hey @asottile,

cygwin seems to set 32kb (https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/include/limits.h;h=3550c4fcb82589062b410b4f9b0c4e5fa09eb2e8;hb=28ecec475ff423b368bcca329f42cfed29308d61#l173)

So, should pre-commit/xargs implementation consider a value between 32k and 128k (+ 4kb minimum) ?

Another refactor in pre-commit/xargs.py is to stop using str length and use the "lenght" in bytes.
makes sense to me, I'd try and follow `xargs` as close as possible!  using `bytes` probably makes sense as well (the limit on windows is actually characters because the api takes wide chars directly, yay)
Here's summary (just for future reference) https://blogs.msdn.microsoft.com/oldnewthing/20031210-00/?p=41553/

I was looking `xargs` code and they do a lot of things to compute the correct value. Maybe we could take a shortcut ?
Something like:
```python
import os
import sys


def _get_max_arg_length():
    headroom = 2 * 1024
    min_arg_length = 4 * 1024
    initial_max_arg_length = 128 * 1024

    if sys.platform == 'win32':
        initial_max_arg_length = 32 * 1024

    max_arg = initial_max_arg_length - sys.getsizeof(os.environ) - headroom
    # xxx: sys.getsizeof just to illustrate the idea

    return max(min_arg_length, max_arg)

```

makes sense to me!   probably use `os.sysconf` on posix to get the `128 * 1024` value but other than that seems about right
Just one more thing. the output of `xargs --show-limits`:
```
POSIX upper limit on argument length (this system): 2088635
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2082166
Size of command buffer we are actually using: 131072
```

So, `SC_ARG_MAX` is 2088635 (200k), should we use the maximum value possible or consider 128k as "safe-limit" ?
I forget how xargs makes its decision but I think the smaller value makes sense (128k)

Resolving patch:
diff --git a/pre_commit/xargs.py b/pre_commit/xargs.py
--- a/pre_commit/xargs.py
+++ b/pre_commit/xargs.py
@@ -5,6 +5,7 @@
 import concurrent.futures
 import contextlib
 import math
+import os
 import sys
 
 import six
@@ -13,10 +14,24 @@
 from pre_commit.util import cmd_output
 
 
-# TODO: properly compute max_length value
-def _get_platform_max_length():
-    # posix minimum
-    return 4 * 1024
+def _environ_size(_env=None):
+    environ = _env if _env is not None else getattr(os, 'environb', os.environ)
+    size = 8 * len(environ)  # number of pointers in `envp`
+    for k, v in environ.items():
+        size += len(k) + len(v) + 2  # c strings in `envp`
+    return size
+
+
+def _get_platform_max_length():  # pragma: no cover (platform specific)
+    if os.name == 'posix':
+        maximum = os.sysconf(str('SC_ARG_MAX')) - 2048 - _environ_size()
+        maximum = min(maximum, 2 ** 17)
+        return maximum
+    elif os.name == 'nt':
+        return 2 ** 15 - 2048  # UNICODE_STRING max - headroom
+    else:
+        # posix minimum
+        return 2 ** 12
 
 
 def _command_length(*cmd):
@@ -52,7 +67,7 @@ def partition(cmd, varargs, target_concurrency, _max_length=None):
     # Reversed so arguments are in order
     varargs = list(reversed(varargs))
 
-    total_length = _command_length(*cmd)
+    total_length = _command_length(*cmd) + 1
     while varargs:
         arg = varargs.pop()
 
@@ -69,7 +84,7 @@ def partition(cmd, varargs, target_concurrency, _max_length=None):
             # We've exceeded the length, yield a command
             ret.append(cmd + tuple(ret_cmd))
             ret_cmd = []
-            total_length = _command_length(*cmd)
+            total_length = _command_length(*cmd) + 1
             varargs.append(arg)
 
     ret.append(cmd + tuple(ret_cmd))
@@ -99,7 +114,7 @@ def xargs(cmd, varargs, **kwargs):
     stderr = b''
 
     try:
-        parse_shebang.normexe(cmd[0])
+        cmd = parse_shebang.normalize_cmd(cmd)
     except parse_shebang.ExecutableNotFoundError as e:
         return e.to_output()
 


Test patch:
diff --git a/tests/xargs_test.py b/tests/xargs_test.py
--- a/tests/xargs_test.py
+++ b/tests/xargs_test.py
@@ -10,9 +10,24 @@
 import pytest
 import six
 
+from pre_commit import parse_shebang
 from pre_commit import xargs
 
 
+@pytest.mark.parametrize(
+    ('env', 'expected'),
+    (
+        ({}, 0),
+        ({b'x': b'1'}, 12),
+        ({b'x': b'12'}, 13),
+        ({b'x': b'1', b'y': b'2'}, 24),
+    ),
+)
+def test_environ_size(env, expected):
+    # normalize integer sizing
+    assert xargs._environ_size(_env=env) == expected
+
+
 @pytest.fixture
 def win32_py2_mock():
     with mock.patch.object(sys, 'getfilesystemencoding', return_value='utf-8'):
@@ -56,7 +71,7 @@ def test_partition_limits():
             '.' * 6,
         ),
         1,
-        _max_length=20,
+        _max_length=21,
     )
     assert ret == (
         ('ninechars', '.' * 5, '.' * 4),
@@ -70,21 +85,21 @@ def test_partition_limit_win32_py3(win32_py3_mock):
     cmd = ('ninechars',)
     # counted as half because of utf-16 encode
     varargs = ('😑' * 5,)
-    ret = xargs.partition(cmd, varargs, 1, _max_length=20)
+    ret = xargs.partition(cmd, varargs, 1, _max_length=21)
     assert ret == (cmd + varargs,)
 
 
 def test_partition_limit_win32_py2(win32_py2_mock):
     cmd = ('ninechars',)
     varargs = ('😑' * 5,)  # 4 bytes * 5
-    ret = xargs.partition(cmd, varargs, 1, _max_length=30)
+    ret = xargs.partition(cmd, varargs, 1, _max_length=31)
     assert ret == (cmd + varargs,)
 
 
 def test_partition_limit_linux(linux_mock):
     cmd = ('ninechars',)
     varargs = ('😑' * 5,)
-    ret = xargs.partition(cmd, varargs, 1, _max_length=30)
+    ret = xargs.partition(cmd, varargs, 1, _max_length=31)
     assert ret == (cmd + varargs,)
 
 
@@ -134,9 +149,9 @@ def test_xargs_smoke():
     assert err == b''
 
 
-exit_cmd = ('bash', '-c', 'exit $1', '--')
+exit_cmd = parse_shebang.normalize_cmd(('bash', '-c', 'exit $1', '--'))
 # Abuse max_length to control the exit code
-max_length = len(' '.join(exit_cmd)) + 2
+max_length = len(' '.join(exit_cmd)) + 3
 
 
 def test_xargs_negate():
@@ -165,14 +180,14 @@ def test_xargs_retcode_normal():
 
 
 def test_xargs_concurrency():
-    bash_cmd = ('bash', '-c')
+    bash_cmd = parse_shebang.normalize_cmd(('bash', '-c'))
     print_pid = ('sleep 0.5 && echo $$',)
 
     start = time.time()
     ret, stdout, _ = xargs.xargs(
         bash_cmd, print_pid * 5,
         target_concurrency=5,
-        _max_length=len(' '.join(bash_cmd + print_pid)),
+        _max_length=len(' '.join(bash_cmd + print_pid)) + 1,
     )
     elapsed = time.time() - start
     assert ret == 0