Issue 1:
Impossible to do parallel numpy computations using python subprocesses,

because just simple import numpy requires ~1Gb paging file

I have 32Gb ram and 4Gb paging file

this code causes an error

import multiprocessing
import time

def proc():
import numpy
time.sleep(1000)

if __name__ == '__main__':
for i in range(50):
multiprocessing.Process(target=proc).start()

time.sleep(1000)

Error message:

File "D:\DevelopPPP\projects\DeepFaceLive\_internal\python\lib\site-packages\numpy\__init__.py", line 140, in <module>
from . import core
File "D:\DevelopPPP\projects\DeepFaceLive\_internal\python\lib\site-packages\numpy\core\__init__.py", line 72, in <module>
from . import numeric
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 674, in exec_module
File "<frozen importlib._bootstrap_external>", line 764, in get_code
File "<frozen importlib._bootstrap_external>", line 833, in get_data
MemoryError

OS : Windows 10 64-bit

NumPy/Python version information:

1.19.5 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]

Issue 2:
Background

We run our compile runs with daemontools to lock virtual memory usage to around 1GB per process (important when running very very parallel builds). During update from 1.19.4 -> 1.19.5 our nightlies trip over code that uses numpy.

Fine with numpy==1.19.4 (1GB):

softlimit -a $(expr 1024 \* 1024 \* 1024) make

Needed to build with numpy==1.19.5 (1.7GB):

softlimit -a $(expr 1740 \* 1024 \* 1024) make

I've whittled it down to the smallest possible reproduction which is simply importing numpy version 1.19.5 with 1GB virtual memory limit. See "Reproducing code example"

To examine further I ratcheted the limit up/down with 1.19.4 and 1.9.5 to find the point at which import succeeds.

version
approx minimum import limit

1.19.4
softlimit -a 400000000 python3

1.19.5
softlimit -a 1200000000 python3

1.20.0rc2
softlimit -a 1200000000 python3

1.19.3
softlimit -a 1200000000 python3

1.19.2
softlimit -a 400000000 python3

Note this is with nothing but import numpy. The overhead difference ~800MB matches the limit changes we needed to get builds running.

Reproducing code example:

You can reproduce this just with loading of the module for version 1.19.5 on ubuntu:20.04 with daemontools. See below for Dockerfile to set up clean 20.04 to reproduce.

import numpy as np

Invoked with softlimit -a 1073741824 python3

root@88e4e56b16a1:~# pip3 install numpy==1.19.5
Collecting numpy==1.19.5
Downloading numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB)
|████████████████████████████████| 14.9 MB 10.7 MB/s
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.19.4
Uninstalling numpy-1.19.4:
Successfully uninstalled numpy-1.19.4
Successfully installed numpy-1.19.5
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 143, in <module>
from . import lib
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py", line 40, in <module>
from .arraypad import *
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py", line 533, in <module>
def pad(array, pad_width, mode='constant', **kwargs):
File "/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py", line 180, in decorator
source_object = compile(
MemoryError
>>>

But ok with 1.19.4:

root@88e4e56b16a1:~# pip3 install numpy==1.19.4
Collecting numpy==1.19.4
Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.19.5
Uninstalling numpy-1.19.5:
Successfully uninstalled numpy-1.19.5
Successfully installed numpy-1.19.4
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

Dockerfile for reproducing - vanilla 20.04 + python3 + pip + softlimit

cat Dockerfile
from ubuntu:20.04

RUN apt-get update && apt-get install -y \
python3 python3-pip \
daemontools \
&& apt-get clean

docker build -t numpylimit .
docker run --rm -it numpylimit /bin/bash

Error message:

>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 143, in <module>
from . import lib
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py", line 40, in <module>
from .arraypad import *
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py", line 533, in <module>
def pad(array, pad_width, mode='constant', **kwargs):
File "/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py", line 180, in decorator
source_object = compile(
MemoryError

NumPy/Python version information:

1.19.5

Let me know if there is any more information I can give you.