Repository: pre-commit/pre-commit

Issue:
Concurrent execution results in uneven work per thread
I'm running `pre-commit` from current `master` to test the concurrency feature introduced with #851. While it in general seems to work, work is distributed pretty uneven. One hook we run is [`prospector`](https://github.com/guykisel/prospector-mirror) which is nice for testing, because it takes a relatively long time and it prints the time taken in its output.

Running `pre-commit run -a --verbose prospector | grep "Time Taken"` on a medium sized project (~100 Python files) results in the following distribution of work to the available 4 logical CPU cores:
```
Time Taken: 17.10 seconds
Time Taken: 8.70 seconds
Time Taken: 18.68 seconds
Time Taken: 108.02 seconds
```

Especially compared to running it with concurrency disabled (using `PRE_COMMIT_NO_CONCURRENCY`), it's pretty obvious that concurrency doesn't provide any real benefit here:
```
Time Taken: 116.95 seconds
```

I'd be happy to help debugging this further. Just tell me what other information you need. :slightly_smiling_face: 

Discussion:

Patch:
diff --git a/pre_commit/languages/helpers.py b/pre_commit/languages/helpers.py
index aa5a5d1..7ab117b 100644
--- a/pre_commit/languages/helpers.py
+++ b/pre_commit/languages/helpers.py
@@ -2,12 +2,18 @@ from __future__ import unicode_literals
 
 import multiprocessing
 import os
+import random
 import shlex
 
+import six
+
 from pre_commit.util import cmd_output
 from pre_commit.xargs import xargs
 
 
+FIXED_RANDOM_SEED = 1542676186
+
+
 def run_setup_cmd(prefix, cmd):
     cmd_output(*cmd, cwd=prefix.prefix_dir, encoding=None)
 
@@ -64,5 +70,21 @@ def target_concurrency(hook):
                 return 1
 
 
+def _shuffled(seq):
+    """Deterministically shuffle identically under both py2 + py3."""
+    fixed_random = random.Random()
+    if six.PY2:  # pragma: no cover (py2)
+        fixed_random.seed(FIXED_RANDOM_SEED)
+    else:
+        fixed_random.seed(FIXED_RANDOM_SEED, version=1)
+
+    seq = list(seq)
+    random.shuffle(seq, random=fixed_random.random)
+    return seq
+
+
 def run_xargs(hook, cmd, file_args):
+    # Shuffle the files so that they more evenly fill out the xargs partitions,
+    # but do it deterministically in case a hook cares about ordering.
+    file_args = _shuffled(file_args)
     return xargs(cmd, file_args, target_concurrency=target_concurrency(hook))


Test patch:
diff --git a/tests/languages/helpers_test.py b/tests/languages/helpers_test.py
index e7bd470..f77c305 100644
--- a/tests/languages/helpers_test.py
+++ b/tests/languages/helpers_test.py
@@ -62,3 +62,7 @@ def test_target_concurrency_cpu_count_not_implemented():
     ):
         with mock.patch.dict(os.environ, {}, clear=True):
             assert helpers.target_concurrency({'require_serial': False}) == 1
+
+
+def test_shuffled_is_deterministic():
+    assert helpers._shuffled(range(10)) == [3, 7, 8, 2, 4, 6, 5, 1, 0, 9]