BUG: pivot/unstack leading to too many items should raise exception
State: closed


- [x] closes #20601
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Hello @sweb! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/core/reshape/reshape.py`](https://github.com/sweb/pandas/blob/5f03406088a94f71f1602aaaeb2273a9b9a7b975/pandas/core/reshape/reshape.py) !

 - There are no PEP8 issues in the file [`pandas/tests/reshape/test_reshape.py`](https://github.com/sweb/pandas/blob/5f03406088a94f71f1602aaaeb2273a9b9a7b975/pandas/tests/reshape/test_reshape.py) !


is this like #20784 ? (you can take that over if you wish)
@jreback Yes, it is quite similar, however my changes are a bit more local. I do not attempt to compute how many rows/cols are allowed but just check whether the already computed value is negative due to an integer overflow.

However, it appears that this is OS dependent. On Unix `np.product` will not result in a negative value, but directly computes with `int64` or promotes it after reaching a limit.

In my current solution I explicitly set `dtype` of `np.product` to `int32` - this will mean a change in current behavior for Unix systems.

Do you think this is a valid approach or should we set it to `int64` as originally proposed?
@sweb the approach in the other issue is good.
@jreback okay. Since I am new here one last quick question: Is it possible for me to incorporate the other PR into this / get the other PR ready for merge or do I have to copy it by hand and push it again? Thanks!
you can simply cherry-pick that commit (ideally). or you can copy it whatever works.
@jreback Thanks for the advice. I patched the commits of the other PR into this branch. Unfortunately, the proposed solution does not solve the problem - at least not on my environment.

Since this is most likely a Windows-specific error (I checked the example from #20601 on MacOs and everything works fine - I did not check it on Linux yet) the proposed solution would actually limit the capabilities of unstack().

In addition, on my Win10 machine, I still get an error, since `num_rows * num_columns` also overflows and the condition `if num_rows * num_columns > (2 ** 31 - 1)` is never met.

I modified the PR in that a numpy warning, indicating the overflow is caught instead of checking the upper bound of int32. The tests are only applied to Windows environments.

Can you replicate this? Maybe my Windows environment setup is screwed up.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=h1) Report
> Merging [#23512](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d5e5bf761092c59eeb9b8750f05f2bc29fb45927?src=pr&el=desc) will **decrease** coverage by `60.41%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23512/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #23512       +/-   ##
===========================================
- Coverage   92.31%   31.89%   -60.42%     
===========================================
  Files         166      166               
  Lines       52412    52426       +14     
===========================================
- Hits        48382    16722    -31660     
- Misses       4030    35704    +31674
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `30.29% <0%> (-60.45%)` | :arrow_down: |
| #single | `31.89% <0%> (-11.17%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/reshape/pivot.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9waXZvdC5weQ==) | `8.77% <ø> (-87.78%)` | :arrow_down: |
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `7.97% <0%> (-91.6%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-98.65%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| ... and [128 more](https://codecov.io/gh/pandas-dev/pandas/pull/23512/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=footer). Last update [d5e5bf7...a3cdbca](https://codecov.io/gh/pandas-dev/pandas/pull/23512?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you merge master
Master was merged!
This should raise regardless of the platform, the OP question is simply too large at 4B entries and is unsupportable.

if you want to merge master and update will look.
The ValueError now raises regardless of OS.
Whatsnew added, thank you for your help!
thanks!