Repository: pre-commit/pre-commit

Issue:
CPU count logic does not distinguish "usable" CPUs inside cgroup'd containers (e.g. inside k8s)
### search you tried in the issue tracker

multiprocessing, k8s, usable, cpu

### describe your issue

When invoking pre-commit from within a container environment with cgroup cpu enforcement, pre-commit fans out more than expected (based on the total number of CPUs rather than the number of _usable_ CPUs). This ends up causing unexpected behaviour and makes it hard to optimise for performance when running hooks in CI inside containers (since [pre-commit's parallelism is non-configurable](https://github.com/pre-commit/pre-commit/issues/1710)), which leaves us with either "shard using too many CPUs" or `require_serial: true`, both of which aren't ideal.

The cause is seemingly due to the implementation in xargs.py: https://github.com/pre-commit/pre-commit/blob/bde292b51078357384602ebe6b9b27b1906987e5/pre_commit/xargs.py#L28 which uses `multiprocessing.cpu_count()`, which is documented to [not [be] equivalent to the number of CPUs the current process can use](https://docs.python.org/3/library/multiprocessing.html#:~:text=This%20number%20is%20not%20equivalent%20to%20the%20number%20of%20CPUs%20the%20current%20process%20can%20use.%20The%20number%20of%20usable%20CPUs%20can%20be%20obtained%20with%20len(os.sched_getaffinity(0))). This is confirmed by running `python3 -c 'import multiprocessing; print(multiprocessing.cpu_count())'` inside the container environment: the number is higher than expected.

From the docs, it looks like a cgroup-compatible way of grabbing the number of usable CPUs would be to use `len(os.sched_getaffinity(0))`, but I don't know if that has undesirable downsides. I also don't know if this would be a disruptive breaking change to anyone relying on the old behaviour, so I wanted to make this issue first to get your thoughts.

### pre-commit --version

pre-commit 2.19.0

### .pre-commit-config.yaml

```yaml
n/a - can provide a dummy config invoking `python3 -c 'import multiprocessing; print(multiprocessing.cpu_count())'` if more info needed, but this isn't specific to any one hook or config option
```


### ~/.cache/pre-commit/pre-commit.log (if present)

_No response_


Discussion:
sched_getaffinity isn't available on other platforms but could probably be used if available

Resolving patch:
diff --git a/pre_commit/xargs.py b/pre_commit/xargs.py
--- a/pre_commit/xargs.py
+++ b/pre_commit/xargs.py
@@ -24,6 +24,14 @@
 
 
 def cpu_count() -> int:
+    try:
+        # On systems that support it, this will return a more accurate count of
+        # usable CPUs for the current process, which will take into account
+        # cgroup limits
+        return len(os.sched_getaffinity(0))
+    except AttributeError:
+        pass
+
     try:
         return multiprocessing.cpu_count()
     except NotImplementedError:


Test patch:
diff --git a/tests/lang_base_test.py b/tests/lang_base_test.py
--- a/tests/lang_base_test.py
+++ b/tests/lang_base_test.py
@@ -30,6 +30,19 @@ def fake_expanduser(pth):
         yield
 
 
+@pytest.fixture
+def no_sched_getaffinity():
+    # Simulates an OS without os.sched_getaffinity available (mac/windows)
+    # https://docs.python.org/3/library/os.html#interface-to-the-scheduler
+    with mock.patch.object(
+        os,
+        'sched_getaffinity',
+        create=True,
+        side_effect=AttributeError,
+    ):
+        yield
+
+
 def test_exe_exists_does_not_exist(find_exe_mck, homedir_mck):
     find_exe_mck.return_value = None
     assert lang_base.exe_exists('ruby') is False
@@ -116,7 +129,17 @@ def test_no_env_noop(tmp_path):
     assert before == inside == after
 
 
-def test_target_concurrency_normal():
+def test_target_concurrency_sched_getaffinity(no_sched_getaffinity):
+    with mock.patch.object(
+        os,
+        'sched_getaffinity',
+        return_value=set(range(345)),
+    ):
+        with mock.patch.dict(os.environ, clear=True):
+            assert lang_base.target_concurrency() == 345
+
+
+def test_target_concurrency_without_sched_getaffinity(no_sched_getaffinity):
     with mock.patch.object(multiprocessing, 'cpu_count', return_value=123):
         with mock.patch.dict(os.environ, {}, clear=True):
             assert lang_base.target_concurrency() == 123
@@ -134,7 +157,7 @@ def test_target_concurrency_on_travis():
         assert lang_base.target_concurrency() == 2
 
 
-def test_target_concurrency_cpu_count_not_implemented():
+def test_target_concurrency_cpu_count_not_implemented(no_sched_getaffinity):
     with mock.patch.object(
             multiprocessing, 'cpu_count', side_effect=NotImplementedError,
     ):