Repository: pre-commit/pre-commit

Issue:
Windows: Large number of files causes `xargs: ... Bad file number`
Originally here: https://github.com/pre-commit/pre-commit-hooks/issues/41



Discussion:
I think I might be able to solve this with xargs -s perhaps

Based on my googling that does sound like a good way to handle this issue. It's actually surprisingly difficult to find much information about this.

The other idea I have is it is hitting the file descriptor limit which seems to be surprisingly low on windows.

Is pre-commit actually opening all of those files at once, though? It doesn't seem like these hooks would actually be hanging on to file descriptors. The list of filenames is just a bunch of strings, after all.

pre-commit shouldn't be grabbing file descriptors.  I did see the same problem with just `git add` which makes me think it might be some git internal that's misbehaving.  At the moment I'm trying to reproduce this under test (and not just interactively) and failing :/

> On computers running Microsoft Windows XP or later, the maximum length of the string that you can use at the command prompt is 8191 characters. On computers running Microsoft Windows 2000 or Windows NT 4.0, the maximum length of the string that you can use at the command prompt is 2047 characters.

In Linux it's more like a few megabytes worth of text I think.

Linux depends on the implementation. POSIX says 4096 is the lowest minimum so I'm going to choose `-s 4000` to be pretty safe.


Resolving patch:
diff --git a/pre_commit/languages/helpers.py b/pre_commit/languages/helpers.py
--- a/pre_commit/languages/helpers.py
+++ b/pre_commit/languages/helpers.py
@@ -10,7 +10,9 @@ def file_args_to_stdin(file_args):
 def run_hook(env, hook, file_args):
     quoted_args = [pipes.quote(arg) for arg in hook['args']]
     return env.run(
-        ' '.join(['xargs', '-0', hook['entry']] + quoted_args),
+        # Use -s 4000 (slightly less than posix mandated minimum)
+        # This is to prevent "xargs: ... Bad file number" on windows
+        ' '.join(['xargs', '-0', '-s4000', hook['entry']] + quoted_args),
         stdin=file_args_to_stdin(file_args),
         retcode=None,
     )


Test patch:
diff --git a/tests/commands/run_test.py b/tests/commands/run_test.py
--- a/tests/commands/run_test.py
+++ b/tests/commands/run_test.py
@@ -305,3 +305,32 @@ def test_get_changed_files():
         '3387edbb1288a580b37fe25225aa0b856b18ad1a',
     )
     assert files == ['CHANGELOG.md', 'setup.py']
+
+
+def test_lots_of_files(mock_out_store_directory, tmpdir_factory):
+    # windows xargs seems to have a bug, here's a regression test for
+    # our workaround
+    git_path = make_consuming_repo(tmpdir_factory, 'python_hooks_repo')
+    with cwd(git_path):
+        # Override files so we run against them
+        with io.open(
+            '.pre-commit-config.yaml', 'a+',
+        ) as config_file:
+            config_file.write('        files: ""\n')
+
+        # Write a crap ton of files
+        for i in range(400):
+            filename = '{0}{1}'.format('a' * 100, i)
+            open(filename, 'w').close()
+
+        cmd_output('bash', '-c', 'git add .')
+        install(Runner(git_path))
+
+        # Don't want to write to home directory
+        env = dict(os.environ, **{'PRE_COMMIT_HOME': tmpdir_factory.get()})
+        cmd_output(
+            'git', 'commit', '-m', 'Commit!',
+            # git commit puts pre-commit to stderr
+            stderr=subprocess.STDOUT,
+            env=env,
+        )