Repository: numpy/numpy
Pull request: BUG: Histogramdd breaks on big arrays in Windows
Resolved the issue with line change from int to np.intp in numpy/numpy/lib/histograms.py
New test function was added called test_big_arrays on numpy/lib/tests/test_histograms.py

<!--         ----------------------------------------------------------------
                MAKE SURE YOUR PR GETS THE ATTENTION IT DESERVES!
                ----------------------------------------------------------------

*  FORMAT IT RIGHT:
      https://www.numpy.org/devdocs/dev/development_workflow.html#writing-the-commit-message

*  IF IT'S A NEW FEATURE OR API CHANGE, TEST THE WATERS:
      https://www.numpy.org/devdocs/dev/development_workflow.html#get-the-mailing-list-s-opinion

*  HIT ALL THE GUIDELINES:
      https://numpy.org/devdocs/dev/index.html#guidelines

*  WHAT TO DO IF WE HAVEN'T GOTTEN BACK TO YOU:
      https://www.numpy.org/devdocs/dev/development_workflow.html#getting-your-pr-reviewed
-->

Resolved #22288 
Pydata Sprint NYC 2022

Problem statement:
BUG: Histogramdd breaks on big arrays in Windows
### Describe the issue:

numpy.histogramdd fails with large arrays on Windows 10 64-bit. This is likely due to a bizarre behavior on Windows64 where the default integer dtype returns as int32 causing numpy.prod to overflow

### Reproduce the code example:

```python
>>> import numpy as np
>>> sample = np.zeros([100000000, 3])
>>> xbins = 400
>>> ybins = 400
>>> zbins = np.arange(16000)
>>> hist = np.histogramdd(sample=sample, bins=(xbins, ybins, zbins))
```


### Error message:

```shell
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<__array_function__ internals>", line 180, in histogramdd
  File "C:\Users\stephanies\AppData\Local\Continuum\miniconda3\envs\numpy_only\lib\site-packages\numpy\lib\histograms.py", line 1095, in histogramdd
    hist = np.bincount(xy, weights, minlength=nbin.prod())
  File "<__array_function__ internals>", line 180, in bincount
ValueError: 'minlength' must not be negative
```


### NumPy/Python version information:

1.23.3 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:29:51) [MSC v.1929 64 bit (AMD64)]

### Context for the issue:

This is a platform issue preventing the generation of large histograms. The same code above works fine on a linux machine

>>> import numpy as np
>>> sample = np.zeros([100000000, 3])
>>> xbins = 400
>>> ybins = 400
>>> zbins = np.arange(16000)
>>> hist = np.histogramdd(sample=sample, bins=(xbins, ybins, zbins))
>>> hist[0].shape
(400, 400, 15999)

Hints from issue discussion:
I think you are right about the cause. Should be easy to fix once the bad code is located.
This would seem to be the culprit; I'd expect that simply changing the datatype to `np.intp` will fix the issue.
https://github.com/numpy/numpy/blob/2524a53ba30c1207770a27a513eb18a33837a145/numpy/lib/histograms.py#L973

Commit message: Update test_histograms.py

Patch to review:
diff --git a/numpy/lib/histograms.py b/numpy/lib/histograms.py
index 704f69dcece7..0dfa7b4c1819 100644
--- a/numpy/lib/histograms.py
+++ b/numpy/lib/histograms.py
@@ -970,7 +970,7 @@ def histogramdd(sample, bins=10, range=None, density=None, weights=None):
         sample = np.atleast_2d(sample).T
         N, D = sample.shape
 
-    nbin = np.empty(D, int)
+    nbin = np.empty(D, np.intp)
     edges = D*[None]
     dedges = D*[None]
     if weights is not None:
diff --git a/numpy/lib/tests/test_histograms.py b/numpy/lib/tests/test_histograms.py
index 87643169b931..cf543301d0bb 100644
--- a/numpy/lib/tests/test_histograms.py
+++ b/numpy/lib/tests/test_histograms.py
@@ -6,6 +6,7 @@
     assert_array_almost_equal, assert_raises, assert_allclose,
     assert_array_max_ulp, assert_raises_regex, suppress_warnings,
     )
+from numpy.testing._private.utils import requires_memory
 import pytest
 
 
@@ -397,6 +398,16 @@ def test_histogram_bin_edges(self):
         edges = histogram_bin_edges(arr, bins='auto', range=(0, 1))
         assert_array_equal(edges, e)
 
+    @requires_memory(free_bytes=100000000 * 3)
+    @pytest.mark.slow
+    def test_big_arrays(self):
+        sample = np.zeros([100000000, 3])
+        xbins = 400
+        ybins = 400
+        zbins = np.arange(16000)
+        hist = np.histogramdd(sample=sample, bins=(xbins, ybins, zbins))
+        assert_equal(type(hist), type((1, 2)))
+
 
 class TestHistogramOptimBinNums:
     """


Code review comments:
diff_hunk: @@ -397,6 +398,16 @@ def test_histogram_bin_edges(self):
         edges = histogram_bin_edges(arr, bins='auto', range=(0, 1))
         assert_array_equal(edges, e)
 
+    @requires_memory(free_bytes=100000000 * 3)
path: numpy/lib/tests/test_histograms.py
text: @user1:
Thanks.  Lets refine this a little bit.  If you check `sample.nbytes`, you will see that it is 8 times larger.   Also the result array (with the current choices) has a significant size.  You could just reduce that size by a lot, even then it probably makes sense to round up a little bit.
(This is a sanity check that we should be good after all.  The decorator also catches `MemoryError` so we should be fine, but I would just round up generously.)

@author:
Sure. That makes sense. I made it 100 times larger

@user1:
OK, if I do the math, this should be about double of what is needed, and 10 GiB are not forbidding these days, so this should be fine.