Issue 1:
Functions like np.zeros, np.empty, np.ones and np.full can be used to allocate large arrays easily. However, they always allocate memory, even if the array contains no data because one of the shape elements is 0. They, in particular, seem to allocate memory ignoring the 0s in the shape (np.product([x for x in shape if x != 0])).

If any of the shape elements in zero, there's no possibility of any memory accesses ever, so the allocation seems unnecessary?

If the shape has a zero, one can get an equivalent array using broadcast_to, which won't allocate.

Reproducing code example:

import numpy as np
import tracemalloc as tm

def test(f):
tm.start()
arr = f((0, 100, 0, 200, 0, 300, 0), dtype=np.uint8)
mem, _ = tm.get_traced_memory()
tm.stop()
print(f"{f.__name__}: allocated memory: {mem} bytes, array size: {arr.size}")

def full(shape, dtype):
return np.full(shape, 123, dtype=dtype)

test(np.zeros)
test(np.empty)
test(np.ones)
test(full)

# workaround, use broadcasting
def broadcast(shape, dtype):
return np.broadcast_to(dtype(), shape)
test(broadcast)

zeros: allocated memory: 6000248 bytes, array size: 0
empty: allocated memory: 6000080 bytes, array size: 0
ones: allocated memory: 6000080 bytes, array size: 0
full: allocated memory: 6000080 bytes, array size: 0
broadcast: allocated memory: 3680 bytes, array size: 0

Numpy/Python version information:

numpy.__version__: 1.18.4

sys.version: 3.6.9 (default, Jul 10 2019, 12:25:55) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)]

Issue 2:
>>> a = np.array([], dtype=np.int64).reshape((6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3))
>>> np.testing.assert_equal(a, a[()])
python(73392,0x113ae6dc0) malloc: can't allocate region
*** mach_vm_map(size=15746400002048) failed (error code=3)
python(73392,0x113ae6dc0) malloc: *** set a breakpoint in malloc_error_break to debug
python(73392,0x113ae6dc0) malloc: can't allocate region
*** mach_vm_map(size=15746400002048) failed (error code=3)
python(73392,0x113ae6dc0) malloc: *** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 349, in assert_equal
return assert_array_equal(actual, desired, err_msg, verbose)
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 936, in assert_array_equal
verbose=verbose, header='Arrays are not equal')
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 770, in assert_array_compare
flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 746, in func_assert_same_pos
raise AssertionError(msg)
AssertionError:
Arrays are not equal

x and y nan location mismatch:
x: array([], shape=(6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3),
dtype=int64)
y: array([], shape=(6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3),
dtype=int64)

I also sometimes get MemoryError from similar examples, for instance, with shape=[7, 2, 6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3], and even np.all gives warnings from malloc.

>>> a = np.array([], dtype=np.int64).reshape([7, 2, 6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3])
>>> np.all(a == a[()])
python(77502,0x10cb79dc0) malloc: can't allocate region
*** mach_vm_map(size=220449600000000) failed (error code=3)
python(77502,0x10cb79dc0) malloc: *** set a breakpoint in malloc_error_break to debug
python(77502,0x10cb79dc0) malloc: can't allocate region
*** mach_vm_map(size=220449600000000) failed (error code=3)
python(77502,0x10cb79dc0) malloc: *** set a breakpoint in malloc_error_break to debug
False

>>> np.testing.assert_equal(a, a[()])
python(77502,0x10cb79dc0) malloc: can't allocate region
*** mach_vm_map(size=220449600000000) failed (error code=3)
python(77502,0x10cb79dc0) malloc: *** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 349, in assert_equal
return assert_array_equal(actual, desired, err_msg, verbose)
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 936, in assert_array_equal
verbose=verbose, header='Arrays are not equal')
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 770, in assert_array_compare
flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')
File "/Users/aaronmeurer/anaconda3/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 728, in func_assert_same_pos
x_id = func(x)
MemoryError: Unable to allocate 0 bytes for an array with shape (7, 2, 6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3) and data type bool

Note:

>>> shape = [7, 2, 6, 10, 6, 10, 0, 5, 1, 9, 10, 2, 6, 2, 6, 0, 10, 9, 5, 1, 5, 1, 10, 3]
>>> np.prod([i for i in shape if i])
220449600000000

It isn't clear to me if this is the sort of thing that should work. I would expect empty arrays to effectively ignore the nonzero entries in the shape except for broadcasting purposes. But maybe there are technical reasons why this cannot happen?

Numpy/Python version information:

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.18.1 3.7.3 | packaged by conda-forge | (default, Mar 27 2019, 15:43:19)
[Clang 4.0.1 (tags/RELEASE_401/final)]