Repository: numpy/numpy
Pull request: ENH: Allow toggling madvise hugepage and fix default
By default this disables madvise hugepage on kernels before 4.6, since
we expect that these typically see large performance regressions when
using hugepages due to slow defragementation code presumably fixed by:

https://github.com/torvalds/linux/commit/7cf91a98e607c2f935dbcc177d70011e95b8faff

This adds support to set the behaviour at startup time through the
``NUMPY_MADVISE_HUGEPAGE`` environment variable.

Fixes gh-15545

----

Still needs some documentation somewhere probably... I am not sure where though, a new site listing all environment variables used during compile or startup time? Relaxed strides, experimental array function protocol, this one, ...?

Problem statement:
many functions slow for bigger-than-cache arrays on linux from numpy>=1.16.5
Performance is badly degraded for many operations, often when array sizes similar to L2/L3 cache size are involved. The effect is present for numpy>=1.16.5 on an Intel i9-9960X Ubuntu 18.04 system, not observed for any numpy version on an Intel i9-8950HK MacOS Mojave system.

### Reproducing code example:

```bash
conda create -n py37np164 python=3.7 numpy=1.16.4
conda create -n py37np165 python=3.7 numpy=1.16.5
conda create -n py37np181 python=3.7 numpy=1.18.1
```
```python
import sys
import numpy as np
from timeit import Timer
print(np.__version__, sys.version)
# results sensitive to hardware cache size
n = 5
sizes = [2**20, 2**21, 2**22, 2**23]
stmts = [
    'np.zeros(({},))',
    'np.random.rand({})',
    'np.linspace(0,1,{})',
    'np.exp(np.zeros(({},)))',
]
for stmt in stmts:
    for size in sizes:
        s = stmt.format(size)
        print(s+':')
        t = Timer(s, globals=globals()).timeit(n)
        print(f'\t{size/n/t} elements/second')
```

output with 1.16.4:
```
1.16.4 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0]
np.zeros((1048576,)):
        61421511.05126992 elements/second
np.zeros((2097152,)):
        88332565.33730087 elements/second
np.zeros((4194304,)):
        27011333457.509125 elements/second
np.zeros((8388608,)):
        55727273740.89583 elements/second
np.random.rand(1048576):
        5537466.835408629 elements/second
np.random.rand(2097152):
        4565838.5126173515 elements/second
np.random.rand(4194304):
        5056596.831366348 elements/second
np.random.rand(8388608):
        4877062.023493181 elements/second
np.linspace(0,1,1048576):
        4726193.511268751 elements/second
np.linspace(0,1,2097152):
        24889336.518190637 elements/second
np.linspace(0,1,4194304):
        10163866.009809423 elements/second
np.linspace(0,1,8388608):
        11765754.105458323 elements/second
np.exp(np.zeros((1048576,))):
        54532313.29674471 elements/second
np.exp(np.zeros((2097152,))):
        33393087.31574934 elements/second
np.exp(np.zeros((4194304,))):
        37155269.74395583 elements/second
np.exp(np.zeros((8388608,))):
        34553576.9698848 elements/second
```

output with 1.16.5:
```
1.16.5 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0]
np.zeros((1048576,)):
        60143502.91423442 elements/second
np.zeros((2097152,)):
        87341513.42070064 elements/second
np.zeros((4194304,)):
        19292941759.91131 elements/second
np.zeros((8388608,)):
        54397048327.81841 elements/second
np.random.rand(1048576):
        5473600.047150185 elements/second
np.random.rand(2097152):
        18752.041155728602 elements/second
np.random.rand(4194304):
        19324.038630861298 elements/second
np.random.rand(8388608):
        14506.20855024289 elements/second
np.linspace(0,1,1048576):
        3856487.073921443 elements/second
np.linspace(0,1,2097152):
        16523085.001294544 elements/second
np.linspace(0,1,4194304):
        11812.122234984105 elements/second
np.linspace(0,1,8388608):
        13185.01070840559 elements/second
np.exp(np.zeros((1048576,))):
        47433268.94274437 elements/second
np.exp(np.zeros((2097152,))):
        107295.34448171785 elements/second
np.exp(np.zeros((4194304,))):
        76042.71390730397 elements/second
np.exp(np.zeros((8388608,))):
        92983.00220850644 elements/second
```

output with 1.18.1:
```
1.18.1 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0]
np.zeros((1048576,)):
        50735117.70738101 elements/second
np.zeros((2097152,)):
        83684116.17091243 elements/second
np.zeros((4194304,)):
        21767035415.033813 elements/second
np.zeros((8388608,)):
        54589913512.271355 elements/second
np.random.rand(1048576):
        8093427.641236661 elements/second
np.random.rand(2097152):
        8188684.253048504 elements/second
np.random.rand(4194304):
        13750.96922032222 elements/second
np.random.rand(8388608):
        12809.948623963204 elements/second
np.linspace(0,1,1048576):
        28700369.02352331 elements/second
np.linspace(0,1,2097152):
        31411688.42840547 elements/second
np.linspace(0,1,4194304):
        12422.915392248138 elements/second
np.linspace(0,1,8388608):
        15747.353944479057 elements/second
np.exp(np.zeros((1048576,))):
        6914276.304259895 elements/second
np.exp(np.zeros((2097152,))):
        9039.82013534727 elements/second
np.exp(np.zeros((4194304,))):
        15507.785512087936 elements/second
np.exp(np.zeros((8388608,))):
        13965.641080531177 elements/second
```

### Python/Numpy version
results of `conda env export` (envs differ only in numpy  and numpy-base versions)
```
name: py37np165
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - blas=1.0=mkl
  - ca-certificates=2020.1.1=0
  - certifi=2019.11.28=py37_0
  - intel-openmp=2020.0=166
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - mkl=2020.0=166
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.0.15=py37ha843d7b_0
  - mkl_random=1.1.0=py37hd6b4f25_0
  - ncurses=6.1=he6710b0_1
  - numpy=1.16.5=py37h7e9f1db_0
  - numpy-base=1.16.5=py37hde5b4d6_0
  - openssl=1.1.1d=h7b6447c_3
  - pip=20.0.2=py37_1
  - python=3.7.6=h0371630_2
  - readline=7.0=h7b6447c_5
  - setuptools=45.1.0=py37_0
  - six=1.14.0=py37_0
  - sqlite=3.31.1=h7b6447c_0
  - tk=8.6.8=hbc83047_0
  - wheel=0.34.2=py37_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
```

### OS/Hardware:
Ubuntu 18.04.3 LTS (GNU/Linux 5.0.0-37-generic x86_64)
```
Handle 0x0061, DMI type 4, 48 bytes
Processor Information
        Socket Designation: LGA 2066 R4
        Type: Central Processor
        Family: Xeon
        Manufacturer: Intel(R) Corporation
        ID: 54 06 05 00 FF FB EB BF
        Signature: Type 0, Family 6, Model 85, Stepping 4
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                PSE-36 (36-bit page size extension)
                CLFSH (CLFLUSH instruction supported)
                DS (Debug store)
                ACPI (ACPI supported)
                MMX (MMX technology supported)
                FXSR (FXSAVE and FXSTOR instructions supported)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                SS (Self-snoop)
                HTT (Multi-threading)
                TM (Thermal monitor supported)
                PBE (Pending break enabled)
        Version: Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz
        Voltage: 1.6 V
        External Clock: 100 MHz
        Max Speed: 4000 MHz
        Current Speed: 3100 MHz
        Status: Populated, Enabled
        Upgrade: Other
        L1 Cache Handle: 0x005E
        L2 Cache Handle: 0x005F
        L3 Cache Handle: 0x0060
        Serial Number: Not Specified
        Asset Tag: UNKNOWN
        Part Number: Not Specified
        Core Count: 16
        Core Enabled: 16
        Thread Count: 32
        Characteristics:
                64-bit capable
                Multi-Core
                Hardware Thread
                Execute Protection
                Enhanced Virtualization
                Power/Performance Control

Handle 0x005E, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L1-Cache
        Configuration: Enabled, Not Socketed, Level 1
        Operational Mode: Write Back
        Location: Internal
        Installed Size: 1024 kB
        Maximum Size: 1024 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Instruction
        Associativity: 8-way Set-associative

Handle 0x005F, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L2-Cache
        Configuration: Enabled, Not Socketed, Level 2
        Operational Mode: Varies With Memory Address
        Location: Internal
        Installed Size: 16384 kB
        Maximum Size: 16384 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Unified
        Associativity: 16-way Set-associative

Handle 0x0060, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L3-Cache
        Configuration: Enabled, Not Socketed, Level 3
        Operational Mode: Varies With Memory Address
        Location: Internal
        Installed Size: 22528 kB
        Maximum Size: 22528 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Unified
        Associativity: Fully Associative
```

Hints from issue discussion:
possibly related to https://github.com/numpy/numpy/pull/14322 somehow?
@victor-shepardson That would be my first guess also. How are you installing numpy? It might be interesting to try a source install if that is not what you are doing, although that might degrade BLAS performance depending on the library, but that isn't what we are looking at here. 
Might look at the conversation in  #14177. What does
`cat /sys/kernel/mm/transparent_hugepage/enabled` show?
Although it is curious that `zeros` doesn't seem to be affected.
I don't see this regression on my hardware (which is apparently much slower than yours).
```
1.18.2.dev0+e0660d4 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]
np.zeros((1048576,)):
	38158780.33276095 elements/second
np.zeros((2097152,)):
	75194525.31342381 elements/second
np.zeros((4194304,)):
	14156793846.309706 elements/second
np.zeros((8388608,)):
	45595218962.68746 elements/second
np.random.rand(1048576):
	5043581.73273908 elements/second
np.random.rand(2097152):
	5065115.222716178 elements/second
np.random.rand(4194304):
	4874188.548954194 elements/second
np.random.rand(8388608):
	4932568.4791796 elements/second
np.linspace(0,1,1048576):
	24176792.05426381 elements/second
np.linspace(0,1,2097152):
	17535581.49985825 elements/second
np.linspace(0,1,4194304):
	15133216.867388567 elements/second
np.linspace(0,1,8388608):
	15924277.461800016 elements/second
np.exp(np.zeros((1048576,))):
	6292804.736660751 elements/second
np.exp(np.zeros((2097152,))):
	5133047.037672353 elements/second
np.exp(np.zeros((4194304,))):
	5169529.959440129 elements/second
np.exp(np.zeros((8388608,))):
	5247382.237551451 elements/second
```
Which is actually a bit faster than 1.16.4. So this is rather curious, and I'd like to know why your numbers can be so much larger. Sounds like something parallel, I wonder if this is a compiler issue.
@charris thanks for the quick reply.

 we've been installing from conda. installing from pip shows same behavior, I'll try building from source tomorrow. 

`/sys/kernel/mm/transparent_hugepage/enabled` reads `always [madvise] never`. I don't understand what that means, why would all three options be present?
@victor-shepardson The options are settable, although I suspect you need to be root. I think there is probably some thrashing going on in order to account for the large drop in performance, but this is getting beyond my experience. @pentschev Thoughts?
This is strange, I wouldn't expect to see any performance difference around CPU cache sizes.

> /sys/kernel/mm/transparent_hugepage/enabled reads always [madvise] never. I don't understand what that means, why would all three options be present?

This means there are 3 options possible, and the one inside square brackets is currently selected. I'm wondering if you could check whether there's a difference when you change that to `always` and `never`. To change that you need root access as @charris mentioned, and can run one of the two below (depending on whether you use `sudo` or not):

```
# as root
echo always > /sys/kernel/mm/transparent_hugepage/enabled

# with sudo
echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
```

If hugepage is really the cause, then I would expect to see that setting it to `never` doesn't present a change in performance in any of the NumPy versions. I'm not sure what to expect from `always`, I'm guessing we'll see the same performance of `madvise`.

Also tagging @hmaarrfk in case there are other ideas I'm not thinking of now.
Can you try to preallocate arrays? That made a big difference for me a while back. Even copying arrays is really slow without preallocation
@charris @pentschev did not have time to try building from source or setting `transparent_hugepage/enabled` today, but discovered something else. The issue appears related to swap/disk cache. We had been fitting models with stochastic gradient descent to large datasets. Data were being streamed continuously from disk, which was keeping disk cache memory and swap full. After killing these jobs, the issue disappeared. We think it was cache/swap specifically, because there was still plenty of available memory, disk bandwidth and idle cpu cores.

Commit message: DOC: Note few global global options we currently have in refdocs

These are options that are controlled typically through environment
variable at startup or compile time.

Patch to review:
diff --git a/doc/release/upcoming_changes/15769.improvement.rst b/doc/release/upcoming_changes/15769.improvement.rst
new file mode 100644
index 000000000000..3f70058f6729
--- /dev/null
+++ b/doc/release/upcoming_changes/15769.improvement.rst
@@ -0,0 +1,15 @@
+Ability to disable madvise hugepages
+------------------------------------
+
+On Linux NumPy has previously added support for madavise
+hugepages which can improve performance for very large arrays.
+Unfortunately, on older Kernel versions this led to peformance
+regressions, thus by default the support has been disabled on
+kernels before version 4.6. To override the default, you can
+use the environment variable::
+
+    NUMPY_MADVISE_HUGEPAGE=0
+
+or set it to 1 to force enabling support. Note that this only makes
+a difference if the operating system is set up to use madvise
+transparent hugepage.
diff --git a/doc/source/reference/global_state.rst b/doc/source/reference/global_state.rst
new file mode 100644
index 000000000000..ac6787b7f9da
--- /dev/null
+++ b/doc/source/reference/global_state.rst
@@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
+threads running at the same time.
+Note that for performant linear algebra NumPy uses a BLAS backend
+such as OpenBLAS or MKL, which may use multiple threads that may
+be controlled by environment variables such as ``OMP_NUM_THREADS``
+depending on what is used.
+One way to control the number of threads is the package
+`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_
+
+
+Madvise Hugepage on Linux
+-------------------------
+
+When working with very large arrays on modern Linux kernels,
+you can experience a significant speedup when transparent
+hugepage is enabled.
+This may always be the case or may use ``madvise`` option as
+seen by::
+
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+
+on most kernels.  When set to ``madvise`` NumPy will typically
+use enable hugepages for a performance boost. This behaviour can
+be set through the environment variable::
+
+    NUMPY_MADVISE_HUGEPAGE=0
+
+or ``1`` for enabling it. When not set, the default is to use
+madvise on Kernels 4.6 and newer. These kernels presumably
+experience a large speedup when set.
+This flag is checked at import time.
+
+
+Interpoerabilty
+===============
+
+The array function protocol which allows array-like objects to
+hook into the NumPy API is currently enabled by default.
+It can be disabled using::
+
+    NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=0
+
+See also `class.__array_function__` for more information.
+This flag is checked at import time.
+
+
+Debugging
+=========
+
+Relaxed Strides Checking
+------------------------
+
+The *compile* time environment variables::
+
+    NPY_RELAXED_STRIDES_DEBUG=0
+    NPY_RELAXED_STRIDES_CHECKING=1
+
+control how NumPy reports contiguity for arrays.
+The default that it is enabled and the debug mode is disabled.
+This setting should always be enabled. Setting the
+debug option can be interesting for testing code written
+in C which iterates through arrays that may or may not be
+contiguous in memory.
+Most users will have no reason to change these, for details
+please see the `memory layout <memory-layout>`_ documentation.
diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst
index 6742d605ac29..3b79fc459ece 100644
--- a/doc/source/reference/index.rst
+++ b/doc/source/reference/index.rst
@@ -22,6 +22,7 @@ For learning how to use NumPy, see also :ref:`user`.
    constants
    ufuncs
    routines
+   global_state
    distutils
    distutils_guide
    c-api/index
diff --git a/numpy/__init__.py b/numpy/__init__.py
index 1d8570f71868..42adfc085a00 100644
--- a/numpy/__init__.py
+++ b/numpy/__init__.py
@@ -286,3 +286,24 @@ def _mac_os_check():
                         error_message))
                 raise RuntimeError(msg)
     del _mac_os_check
+
+    # We usually use madvise hugepages support, but on some old kernels it
+    # is slow and thus better avoided.
+    # Specifically kernel version 4.6 had a bug fix which probably fixed this:
+    # https://github.com/torvalds/linux/commit/7cf91a98e607c2f935dbcc177d70011e95b8faff
+    import os
+    use_hugepage = os.environ.get("NUMPY_MADVISE_HUGEPAGE", None)
+    if sys.platform == "linux" and use_hugepage is None:
+        use_hugepage = 1
+        kernel_version = os.uname().release.split(".")[:2]
+        kernel_version = tuple(int(v) for v in kernel_version)
+        if kernel_version < (4, 6):
+            use_hugepage = 0
+    elif use_hugepage is None:
+        # This is not Linux, so it should not matter, just enable anyway
+        use_hugepage = 1
+    else:
+        use_hugepage = int(use_hugepage)
+
+    # Note that this will currently only make a difference on Linux
+    core.multiarray._multiarray_umath._set_madvise_hugepage(use_hugepage)
diff --git a/numpy/core/src/multiarray/alloc.c b/numpy/core/src/multiarray/alloc.c
index c2b7e9ca7d75..9d11608f91cf 100644
--- a/numpy/core/src/multiarray/alloc.c
+++ b/numpy/core/src/multiarray/alloc.c
@@ -47,6 +47,25 @@ typedef struct {
 static cache_bucket datacache[NBUCKETS];
 static cache_bucket dimcache[NBUCKETS_DIM];
 
+static int _madvise_hugepage = 1;
+
+
+NPY_NO_EXPORT PyObject *
+_set_madvise_hugepage(PyObject *NPY_UNUSED(self), PyObject *enabled_obj)
+{
+    int was_enabled = _madvise_hugepage;
+    int enabled = PyObject_IsTrue(enabled_obj);
+    if (enabled < 0) {
+        return NULL;
+    }
+    _madvise_hugepage = enabled;
+    if (was_enabled) {
+        Py_RETURN_TRUE;
+    }
+    Py_RETURN_FALSE;
+}
+
+
 /* as the cache is managed in global variables verify the GIL is held */
 
 /*
@@ -75,7 +94,7 @@ _npy_alloc_cache(npy_uintp nelem, npy_uintp esz, npy_uint msz,
 #endif
 #ifdef NPY_OS_LINUX
         /* allow kernel allocating huge pages for large arrays */
-        if (NPY_UNLIKELY(nelem * esz >= ((1u<<22u)))) {
+        if (NPY_UNLIKELY(nelem * esz >= ((1u<<22u))) && _madvise_hugepage) {
             npy_uintp offset = 4096u - (npy_uintp)p % (4096u);
             npy_uintp length = nelem * esz - offset;
             /**
diff --git a/numpy/core/src/multiarray/alloc.h b/numpy/core/src/multiarray/alloc.h
index 2b69efc35205..15e31ebb5f2f 100644
--- a/numpy/core/src/multiarray/alloc.h
+++ b/numpy/core/src/multiarray/alloc.h
@@ -6,6 +6,9 @@
 
 #define NPY_TRACE_DOMAIN 389047
 
+NPY_NO_EXPORT PyObject *
+_set_madvise_hugepage(PyObject *NPY_UNUSED(self), PyObject *enabled_obj);
+
 NPY_NO_EXPORT void *
 npy_alloc_cache(npy_uintp sz);
 
diff --git a/numpy/core/src/multiarray/multiarraymodule.c b/numpy/core/src/multiarray/multiarraymodule.c
index 7792fcdcbf88..6411ac0a3864 100644
--- a/numpy/core/src/multiarray/multiarraymodule.c
+++ b/numpy/core/src/multiarray/multiarraymodule.c
@@ -34,6 +34,7 @@
 NPY_NO_EXPORT int NPY_NUMUSERTYPES = 0;
 
 /* Internal APIs */
+#include "alloc.h"
 #include "arrayfunction_override.h"
 #include "arraytypes.h"
 #include "arrayobject.h"
@@ -3971,6 +3972,7 @@ normalize_axis_index(PyObject *NPY_UNUSED(self), PyObject *args, PyObject *kwds)
     return PyInt_FromLong(axis);
 }
 
+
 static struct PyMethodDef array_module_methods[] = {
     {"_get_implementing_args",
         (PyCFunction)array__get_implementing_args,
@@ -4159,6 +4161,8 @@ static struct PyMethodDef array_module_methods[] = {
         METH_VARARGS, NULL},
     {"_add_newdoc_ufunc", (PyCFunction)add_newdoc_ufunc,
         METH_VARARGS, NULL},
+    {"_set_madvise_hugepage", (PyCFunction)_set_madvise_hugepage,
+        METH_O, "Toggle and return madvise hugepage (no OS support check)."},
     {NULL, NULL, 0, NULL}                /* sentinel */
 };
 


Code review comments:
diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
during function calls, however it does support multiple Python
```

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
+threads running at the same time.
+Note that for performant linear algebra NumPy uses a BLAS backend
+such as OpenBLAS or MKL, which may use multiple threads that may
+be controlled by environment variables such as ``OMP_NUM_THREADS``
+depending on what is used.
+One way to control the number of threads is the package
+`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_
+
+
+Madvise Hugepage on Linux
+-------------------------
+
+When working with very large arrays on modern Linux kernels,
+you can experience a significant speedup when transparent
+hugepage is enabled.
+This may always be the case or may use ``madvise`` option as
+seen by::
+
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+
+on most kernels.  When set to ``madvise`` NumPy will typically
+use enable hugepages for a performance boost. This behaviour can
+be set through the environment variable::
+
+    NUMPY_MADVISE_HUGEPAGE=0
+
+or ``1`` for enabling it. When not set, the default is to use
+madvise on Kernels 4.6 and newer. These kernels presumably
+experience a large speedup when set.
+This flag is checked at import time.
+
+
+Interpoerabilty
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
Interoperability
```
Could even say "Interoperability-related Options to be consistent with "Performance-related Options"

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
+threads running at the same time.
+Note that for performant linear algebra NumPy uses a BLAS backend
+such as OpenBLAS or MKL, which may use multiple threads that may
+be controlled by environment variables such as ``OMP_NUM_THREADS``
+depending on what is used.
+One way to control the number of threads is the package
+`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_
+
+
+Madvise Hugepage on Linux
+-------------------------
+
+When working with very large arrays on modern Linux kernels,
+you can experience a significant speedup when transparent
+hugepage is enabled.
+This may always be the case or may use ``madvise`` option as
+seen by::
+
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+
+on most kernels.  When set to ``madvise`` NumPy will typically
+use enable hugepages for a performance boost. This behaviour can
+be set through the environment variable::
+
+    NUMPY_MADVISE_HUGEPAGE=0
+
+or ``1`` for enabling it. When not set, the default is to use
+madvise on Kernels 4.6 and newer. These kernels presumably
+experience a large speedup when set.
+This flag is checked at import time.
+
+
+Interpoerabilty
+===============
+
+The array function protocol which allows array-like objects to
+hook into the NumPy API is currently enabled by default.
+It can be disabled using::
+
+    NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=0
+
+See also `class.__array_function__` for more information.
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
See also :py:meth:`numpy.class.__array_function__` for more information.
```
I think that will make the link work

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
+threads running at the same time.
+Note that for performant linear algebra NumPy uses a BLAS backend
+such as OpenBLAS or MKL, which may use multiple threads that may
+be controlled by environment variables such as ``OMP_NUM_THREADS``
+depending on what is used.
+One way to control the number of threads is the package
+`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_
+
+
+Madvise Hugepage on Linux
+-------------------------
+
+When working with very large arrays on modern Linux kernels,
+you can experience a significant speedup when transparent
+hugepage is enabled.
+This may always be the case or may use ``madvise`` option as
+seen by::
+
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+
+on most kernels.  When set to ``madvise`` NumPy will typically
+use enable hugepages for a performance boost. This behaviour can
+be set through the environment variable::
+
+    NUMPY_MADVISE_HUGEPAGE=0
+
+or ``1`` for enabling it. When not set, the default is to use
+madvise on Kernels 4.6 and newer. These kernels presumably
+experience a large speedup when set.
+This flag is checked at import time.
+
+
+Interpoerabilty
+===============
+
+The array function protocol which allows array-like objects to
+hook into the NumPy API is currently enabled by default.
+It can be disabled using::
+
+    NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=0
+
+See also `class.__array_function__` for more information.
+This flag is checked at import time.
+
+
+Debugging
path: doc/source/reference/global_state.rst
text: @user1:
Same suggestion as above - maybe "Debugging-related Options" to be consistent

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
+threads running at the same time.
+Note that for performant linear algebra NumPy uses a BLAS backend
+such as OpenBLAS or MKL, which may use multiple threads that may
+be controlled by environment variables such as ``OMP_NUM_THREADS``
+depending on what is used.
+One way to control the number of threads is the package
+`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_
+
+
+Madvise Hugepage on Linux
+-------------------------
+
+When working with very large arrays on modern Linux kernels,
+you can experience a significant speedup when transparent
+hugepage is enabled.
+This may always be the case or may use ``madvise`` option as
+seen by::
+
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+
+on most kernels.  When set to ``madvise`` NumPy will typically
+use enable hugepages for a performance boost. This behaviour can
+be set through the environment variable::
+
+    NUMPY_MADVISE_HUGEPAGE=0
+
+or ``1`` for enabling it. When not set, the default is to use
+madvise on Kernels 4.6 and newer. These kernels presumably
+experience a large speedup when set.
+This flag is checked at import time.
+
+
+Interpoerabilty
+===============
+
+The array function protocol which allows array-like objects to
+hook into the NumPy API is currently enabled by default.
+It can be disabled using::
+
+    NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=0
+
+See also `class.__array_function__` for more information.
+This flag is checked at import time.
+
+
+Debugging
+=========
+
+Relaxed Strides Checking
+------------------------
+
+The *compile* time environment variables::
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
The *compile-time* environment variables::
```

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
.. _global_state:
```

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
Performance-Related Options
```

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
NumPy has a few import-time, compile-time, or runtime options
```
Also I'm not very clear on what a startup time option is, do you mean at import? If so, would import-time work?

@author:
Yeah, import time.

diff_hunk: @@ -0,0 +1,85 @@
+.. _globale_state:
+
+************
+Global State
+************
+
+NumPy has a few startup time, compile, or runtime options
+which change the global behaviour.
+Most of these are related to performance or for debugging
+purposes and will not be interesting to the vast majority
+of users.
+
+
+Performance Related Options
+===========================
+
+Number of Threads used for Linear Algebra
+-----------------------------------------
+
+NumPy itself is normally intentionally limited to a single thread
+during function calls, however it does support multiple python
+threads running at the same time.
+Note that for performant linear algebra NumPy uses a BLAS backend
+such as OpenBLAS or MKL, which may use multiple threads that may
+be controlled by environment variables such as ``OMP_NUM_THREADS``
+depending on what is used.
+One way to control the number of threads is the package
+`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_
+
+
+Madvise Hugepage on Linux
+-------------------------
+
+When working with very large arrays on modern Linux kernels,
+you can experience a significant speedup when transparent
+hugepage is enabled.
+This may always be the case or may use ``madvise`` option as
+seen by::
+
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+
+on most kernels.  When set to ``madvise`` NumPy will typically
+use enable hugepages for a performance boost. This behaviour can
path: doc/source/reference/global_state.rst
text: @user1:
```suggestion
use hugepages for a performance boost. This behaviour can
```