Repository: dask/distributed

Issue:
cpu cores estimate based on cgroups
I was reading the announcement for 2.4.0 and got interested in https://github.com/dask/distributed/pull/3039 by @jcrist 

That did lead me to this part of the code:
https://github.com/dask/distributed/blob/7d017c467590c758fa4b8cb2b1193205fe5aa7ad/distributed/system.py#L62

Just by looking at it (and although I'm not an expert I think I know what's going on), I have to observations -- half way between a question and a bugreport.

1. in my docker environment I have here (ubuntu 18.04), the filename is different:

```
~$ cat /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us 
100000
~$ cat /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us 
220000
```
in the code is that path:
```
~$ ls /sys/fs/cgroup/cpuacct,cpu/cpu.cfs_quota_us
ls: cannot access '/sys/fs/cgroup/cpuacct,cpu/cpu.cfs_quota_us': No such file or directory
```

2. The actual calculation is `int(quota / period)`. I think this should round up, not down. The point is, if you have a fraction like "2.5 cores", it will report 2 cores and keep half a core unused. It would be better if it reports 3 cores and then cgroups limits to the actual 2.5.




Discussion:
Thanks for filing the issue. Upon further looking, it looks like OSs are inconsistent about the file path here, both `cpu,cpuacct` and `cpuacct,cpu` are possibilities - we should check for both locations. In regards to your second point, I agree we should probably use `ceil` here.

If you feel like submitting a PR for this, it would be most welcome. Otherwise I'll get to it next week.

Resolving patch:
diff --git a/distributed/system.py b/distributed/system.py
--- a/distributed/system.py
+++ b/distributed/system.py
@@ -1,3 +1,4 @@
+import math
 import os
 import sys
 
@@ -61,16 +62,20 @@ def cpu_count():
 
     # Check cgroups if available
     if sys.platform == "linux":
-        try:
-            with open("/sys/fs/cgroup/cpuacct,cpu/cpu.cfs_quota_us") as f:
-                quota = int(f.read())
-            with open("/sys/fs/cgroup/cpuacct,cpu/cpu.cfs_period_us") as f:
-                period = int(f.read())
-            cgroups_count = int(quota / period)
-            if cgroups_count > 0:
-                count = min(count, cgroups_count)
-        except Exception:
-            pass
+        # The directory name isn't standardized across linux distros, check both
+        for dirname in ["cpuacct,cpu", "cpu,cpuacct"]:
+            try:
+                with open("/sys/fs/cgroup/%s/cpu.cfs_quota_us" % dirname) as f:
+                    quota = int(f.read())
+                with open("/sys/fs/cgroup/%s/cpu.cfs_period_us" % dirname) as f:
+                    period = int(f.read())
+                # We round up on fractional CPUs
+                cgroups_count = math.ceil(quota / period)
+                if cgroups_count > 0:
+                    count = min(count, cgroups_count)
+                break
+            except Exception:
+                pass
 
     return count
 


Test patch:
diff --git a/distributed/tests/test_system.py b/distributed/tests/test_system.py
--- a/distributed/tests/test_system.py
+++ b/distributed/tests/test_system.py
@@ -1,4 +1,7 @@
+import builtins
+import io
 import os
+import sys
 
 import psutil
 import pytest
@@ -13,6 +16,44 @@ def test_cpu_count():
     assert count >= 1
 
 
+@pytest.mark.parametrize("dirname", ["cpuacct,cpu", "cpu,cpuacct", None])
+def test_cpu_count_cgroups(dirname, monkeypatch):
+    def mycpu_count():
+        # Absurdly high, unlikely to match real value
+        return 250
+
+    monkeypatch.setattr(os, "cpu_count", mycpu_count)
+
+    class MyProcess(object):
+        def cpu_affinity(self):
+            # No affinity set
+            return []
+
+    monkeypatch.setattr(psutil, "Process", MyProcess)
+
+    if dirname:
+        paths = {
+            "/sys/fs/cgroup/%s/cpu.cfs_quota_us" % dirname: io.StringIO("2005"),
+            "/sys/fs/cgroup/%s/cpu.cfs_period_us" % dirname: io.StringIO("10"),
+        }
+        builtin_open = builtins.open
+
+        def myopen(path, *args, **kwargs):
+            if path in paths:
+                return paths.get(path)
+            return builtin_open(path, *args, **kwargs)
+
+        monkeypatch.setattr(builtins, "open", myopen)
+        monkeypatch.setattr(sys, "platform", "linux")
+
+    count = cpu_count()
+    if dirname:
+        # Rounds up
+        assert count == 201
+    else:
+        assert count == 250
+
+
 def test_memory_limit():
     limit = memory_limit()
     assert isinstance(limit, int)
@@ -20,6 +61,22 @@ def test_memory_limit():
     assert limit >= 1
 
 
+def test_memory_limit_cgroups(monkeypatch):
+    builtin_open = builtins.open
+
+    def myopen(path, *args, **kwargs):
+        if path == "/sys/fs/cgroup/memory/memory.limit_in_bytes":
+            # Absurdly low, unlikely to match real value
+            return io.StringIO("20")
+        return builtin_open(path, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "open", myopen)
+    monkeypatch.setattr(sys, "platform", "linux")
+
+    limit = memory_limit()
+    assert limit == 20
+
+
 def test_rlimit():
     resource = pytest.importorskip("resource")