BUG: fix construction of Series from dict with nested lists
State: closed

- [x] closes #18625
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=h1) Report
> Merging [#18626](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2c903d594299b2441d4742e777a10e8c76557386?src=pr&el=desc) will **decrease** coverage by `0.17%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18626/graphs/tree.svg?token=eZ4WkYLtcO&height=150&width=650&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18626      +/-   ##
==========================================
- Coverage   91.59%   91.42%   -0.18%     
==========================================
  Files         155      157       +2     
  Lines       51255    51457     +202     
==========================================
+ Hits        46949    47046      +97     
- Misses       4306     4411     +105
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.29% <100%> (-0.15%)` | :arrow_down: |
| #single | `40.6% <75%> (-0.18%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.82% <100%> (+0.01%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `80.39% <0%> (-9.74%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `63.44% <0%> (-1.82%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.18% <0%> (-0.59%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.81% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/sparse/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2FwaS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/compat/openpyxl\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9vcGVucHl4bF9jb21wYXQucHk=) | `90.9% <0%> (ø)` | |
| [pandas/core/sparse/list.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2xpc3QucHk=) | `97.1% <0%> (ø)` | |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `84.48% <0%> (+0.05%)` | :arrow_up: |
| ... and [3 more](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=footer). Last update [2c903d5...c5015c1](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=h1) Report
> Merging [#18626](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4efb39f01f5880122fa38d91e12d217ef70fad9e?src=pr&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18626/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18626      +/-   ##
==========================================
+ Coverage   91.82%   91.85%   +0.03%     
==========================================
  Files         152      152              
  Lines       49255    49252       -3     
==========================================
+ Hits        45226    45239      +13     
+ Misses       4029     4013      -16
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.23% <100%> (+0.03%)` | :arrow_up: |
| #single | `41.91% <90%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.07% <100%> (+0.17%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `84.73% <0%> (+0.2%)` | :arrow_up: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18626/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `66.81% <0%> (+1.73%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=footer). Last update [4efb39f...14f0443](https://codecov.io/gh/pandas-dev/pandas/pull/18626?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The new version scans nested types once rather than twice, but there is room for improvements, which I will push tomorrow.
can u run the series construction asv for this
we may not have enough coverage of the various cases as well
> can u run the series construction asv for this

```
       before           after         ratio
     [fdba1333]       [b5225ee5]
-         263±4ms        235±0.6ms     0.89  series_methods.series_constructor_no_data_datetime_index.time_series_constructor_no_data_datetime_index
-     1.67±0.02ms      1.41±0.02ms     0.84  series_methods.series_value_counts.time_value_counts_int64

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```
(but they are borderline, they don't appear on every run)
can u show the series constructions ones
i don’t thing we cover other than the basic cases
> can u show the series constructions ones

Sorry, I had missed ``ctors.py``, here is the result:
```
     [fdba1333]       [b5225ee5]
-     13.8±0.07μs      12.4±0.08μs     0.90  ctors.Constructors.time_series_from_ndarray
-     11.0±0.03μs      9.92±0.05μs     0.90  ctors.Constructors.time_dtindex_from_series
-        25.2±3μs      22.3±0.05μs     0.88  ctors.Constructors.time_dtindex_from_index_with_series

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```

These are from ``frame_ctor.py`` instead:
```
       before           after         ratio
     [fdba1333]       [b5225ee5]
-     3.98±0.02ms      3.61±0.01ms     0.91  frame_ctor.FromDicts.time_series_ctor_from_dict
-      76.0±0.4ms       68.3±0.2ms     0.90  frame_ctor.FromDicts.time_frame_ctor_nested_dict_int64
-      33.6±0.2ms       30.1±0.1ms     0.90  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('QuarterBegin', 1)
-      30.5±0.4ms       27.3±0.2ms     0.90  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('CDay', 2)
-      30.4±0.2ms      27.1±0.08ms     0.89  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('CustomBusinessDay', 2)
-      30.8±0.1ms       27.4±0.1ms     0.89  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('BusinessDay', 1)
-      30.6±0.3ms      27.1±0.07ms     0.89  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('CDay', 1)
-      30.7±0.2ms      27.0±0.07ms     0.88  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('BDay', 2)
-      31.3±0.2ms       27.5±0.3ms     0.88  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('Day', 2)
-      32.0±0.4ms       28.0±0.5ms     0.87  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('CBMonthEnd', 1)
-      32.2±0.5ms       27.8±0.3ms     0.86  frame_ctor.FromDictwithTimestampOffsets.time_frame_ctor('Day', 1)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```
those benchmarks are mostly not on point. e.g. to we have ones which test list-of-list, dict-of-list, dict-of-array, dict-of-series etc. for frame

list, tuple, dict, array, Index, Series for series
> those benchmarks are mostly not on point. e.g. to we have ones which test

For frames there's the entire ``frame_ctor.py``, but for Series all I could find was ``series_constructor_dict_data_datetime_index`` in ``series_methods.py`` and ``time_series_from_ndarray`` in ``ctors.py``. They just didn't exhibit any significant change.
For the change of list of one element (previously broadcasted, now error), I would do this as deprecation instead of a hard breaking change. Thoughts? 
(it is quite easy to do for us I think, and if even Tom writes code using that .. :-))
> (it is quite easy to do for us I think, and if even Tom writes code using that .. :-))

You mean the tests? They look to me as accidental uses of this "feature" - after all, just passing the scalar is simpler.

Anyway, I'm a bit against deprecation in this case because the docs are quite clear, but sure, it can be done.
> They look to me as accidental uses of this "feature" - after all, just passing the scalar is simpler.

Yes, accidental, but many people can have done it "accidental", and since it works perfectly there was no incentive to fix your code.
> For frames there's the entire frame_ctor.py, but for Series all I could find was series_constructor_dict_data_datetime_index in series_methods.py and time_series_from_ndarray in ctors.py. They just didn't exhibit any significant change.

ok, as indicated above, pls add some.
rebase and will have a look
@toobaz status of this?
> @toobaz status of this?

I'll rebase on #18600 when merged
Rebased.

We need to decide what to do about the "length 1 = scalar" thing. I thought we could just break it since it was never documented, is conceptually wrong, and supporting it complicates our code. Joris suggests a deprecation cycle instead. Could a compromise make sense, in which we remove support but _also_ insert (for a couple of releases) an informative warning saying "broadcasting [...] was never officially supported, but it worked accidentally, support was dropped in version 0.23, replace EXAMPLE_CODE with FIXED_EXAMPLE")?
we already changed the length 1 = scalar iirc

are you suggesting we should revert and deprecate instead?
> we already changed the length 1 = scalar iirc

Sorry, I got confused. True, we have disabled ``pd.Series([1], index=range(3))``. What I should have referred to is ``pd.DataFrame([[1]], index=range(3))``, which still works, and this PR disables.

However, the thing is slightly more complicated than I thought. My idea was that (in analogy with the Series case) ``pd.DataFrame([[1]], index=range(3))`` should have been replaced with ``pd.DataFrame([1], index=range(3))`` (as in "broadcast if you have lower dimensional value"). However, that won't work, because ``[1]`` gets aligned vertically. And by the way, this "feature" is also broken, since ``pd.DataFrame([1, 2, 3], index=range(3))`` works (but could be simply be written as ``pd.DataFrame([[1], [2], [3]], index=range(3))``), while ``pd.DataFrame([1, 2, 3], columns=20)`` (which would be the useful case) doesn't.

So, ideal world: if a lower dim object ``obj`` is passed as data, it is interpreted as ``[obj] * len(index)``, both in ``Series`` and in ``DataFrame``.

Acceptable world: ``Series`` remain as they are, but a 1d list ``obj`` passed as data for a ``DataFrame`` is interpreted as ``[[i]*len(columns) for i in obj]``.

If the ideal world is too disruptive (backward incompatible), we can at least aim for the acceptable one. In this case, I'm no more sure we want to drop support for ``pd.DataFrame([[1]], index=range(3))``.
> However, the thing is slightly more complicated than I thought.

Answered myself in #20837 . Will update this PR accordingly (not in time for 0.23.0).
can you rebase & move note to 0.24
can you rebase
closing as stale, but pls reopen if can continue