Repository: sagemath/sage

Issue:
make sage.parallel.ncpus.ncpus() use os.cpu_count()
<div id="comment:0"></div>

Currently, `sage.parallel.ncpus.ncpus()` uses platform-specific code to determine the number of available CPUs for some specific systems. This functionality is now available in the standard `os` module as `cpu_count()`.

Component: **misc**

Author: **Lorenz Panny**

Branch/Commit: **[public/change_ncpus_to_os_module](https://github.com/sagemath/sagetrac-mirror/tree/public/change_ncpus_to_os_module) @ [`a509210`](https://github.com/sagemath/sagetrac-mirror/commit/a509210125fc50baf72dcb7f2248e96cddf61c8f)**

_Issue created by migration from https://trac.sagemath.org/ticket/34328_




Discussion:
<div id="comment:2" align="right">comment:2</div>

From https://docs.python.org/3/library/os.html#os.cpu_count it sounds like using `os.sched_getaffinity` would be a better choice when available. (It's not available on macOS.)
Changed commit from **[`2ca1cdb`](https://github.com/sagemath/sagetrac-mirror/commit/2ca1cdba6deadc635b1e5cabcab8e02be888a99f)** to **[`a509210`](https://github.com/sagemath/sagetrac-mirror/commit/a509210125fc50baf72dcb7f2248e96cddf61c8f)**
<div id="comment:3"></div>

Branch pushed to git repo; I updated commit sha1. New commits:
<table><tr><td><a href="https://github.com/sagemath/sagetrac-mirror/commit/a509210125fc50baf72dcb7f2248e96cddf61c8f"><code>a509210</code></a></td><td><code>prefer os.sched_getaffinity() when available</code></td></tr></table>

<div id="comment:4" align="right">comment:4</div>

Done.

However, here's an issue: In reality, the `ncpus()` function actually always returns `1` when `SAGE_NUM_THREADS` is not set. This is because `sage-num-threads.py`, which is called by `sage-env`, defaults to returning `1` if `SAGE_NUM_THREADS` isn't already in the environment.

Is this intentional? If yes: Should multi-threading really be disabled by default in 2022?
<div id="comment:5" align="right">comment:5</div>

Previous discussions:

- #25779: sage.parallel.ncpus.ncpus() reports wrong number of cores
- #24937: `@`parallel only uses one thread if SAGE_NUM_THREADS is not set; ncpus is wrong
- #23713: Support SAGE_NUM_THREADS everywhere for parallellism
- [Ask Sage question 42439: `@`parallel: how to use all cpus](https://ask.sagemath.org/question/42439)
- [2019-03 sage-devel discussion: How would you like your parallel linear algebra?](https://groups.google.com/g/sage-devel/c/ZNcijpPhWuc)
- [2018-07 sage-devel discussion: How parallel should `@`parallel be?](https://groups.google.com/g/sage-devel/c/z45LTwQWxTc)

The three tickets above were revealed by this trac query:

- https://trac.sagemath.org/query?order=id&desc=1&summary=~ncpu

and the other discussions were linked to from one of them.

The documentation could clarify what choices were made and why.
<div id="comment:6" align="right">comment:6</div>

Also #33317 Unify computation of number of parallel jobs
Branch has merge conflicts

Resolving patch:
diff --git a/src/sage/parallel/ncpus.py b/src/sage/parallel/ncpus.py
--- a/src/sage/parallel/ncpus.py
+++ b/src/sage/parallel/ncpus.py
@@ -1,46 +1,19 @@
 """
 CPU Detection
 """
-# Parallel Python Software: http://www.parallelpython.com
-# Copyright (c) 2005-2008, Vitalii Vanovschi
-# All rights reserved.
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#    * Redistributions of source code must retain the above copyright notice,
-#      this list of conditions and the following disclaimer.
-#    * Redistributions in binary form must reproduce the above copyright
-#      notice, this list of conditions and the following disclaimer in the
-#      documentation and/or other materials provided with the distribution.
-#    * Neither the name of the author nor the names of its contributors
-#      may be used to endorse or promote products derived from this software
-#      without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
-# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
-# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
-# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
-# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
-# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
-# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
-# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
-# THE POSSIBILITY OF SUCH DAMAGE.
-
-######
-# This is from ParallelPython (the pp.py file).
 
 import os
-import subprocess
 
 
 def ncpus():
     """
-    Detects the number of effective CPUs in the system.
+    Return the number of available CPUs in the system.
+
+    ALGORITHM: :func:`os.sched_getaffinity` or :func:`os.cpu_count`
 
     EXAMPLES::
 
-        sage: sage.parallel.ncpus.ncpus()  # random output -- depends on machine.
+        sage: sage.parallel.ncpus.ncpus()  # random output -- depends on machine
         2
     """
     # Support Sage environment variable SAGE_NUM_THREADS
@@ -53,25 +26,9 @@ def ncpus():
     else:
         return int(n)
 
-    # for Linux, Unix and MacOS
-    if hasattr(os, "sysconf"):
-        if "SC_NPROCESSORS_ONLN" in os.sysconf_names:
-            # Linux and Unix
-            ncpus = os.sysconf("SC_NPROCESSORS_ONLN")
-            if isinstance(ncpus, int) and ncpus > 0:
-                return ncpus
-        else:
-            # MacOS X
-            # deprecated: return int(os.popen2("sysctl -n hw.ncpu")[1].read())
-            process = subprocess.Popen("sysctl -n hw.ncpu", shell=True,
-                                       stdin=subprocess.PIPE,
-                                       stdout=subprocess.PIPE,
-                                       stderr=subprocess.PIPE, close_fds=True)
-            return int(process.stdout.read())
-    # for Windows
-    if "NUMBER_OF_PROCESSORS" in os.environ:
-        ncpus = int(os.environ["NUMBER_OF_PROCESSORS"])
-        if ncpus > 0:
-            return ncpus
-    # return the default value
-    return 1
+    n = None
+
+    if hasattr(os, 'sched_getaffinity'):
+        n = len(os.sched_getaffinity(0))
+
+    return n or os.cpu_count() or 1


Test patch: