Construction of Series from dict containing NaN as key
State: closed

- [x] closes #18480
closes #18515

- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is also a prerequisite for fixing #18455 (which is a prerequisite for fixing #18460 ). The workaround to #18485 is annoying, but it is easy to remove it when the bug is fixed.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=h1) Report
> Merging [#18496](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/38f41e64f4b8a0479f8835022af5e7343ccf8498?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `93.18%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18496/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18496      +/-   ##
==========================================
+ Coverage    91.3%   91.31%   +<.01%     
==========================================
  Files         163      163              
  Lines       49781    49817      +36     
==========================================
+ Hits        45451    45488      +37     
+ Misses       4330     4329       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.11% <93.18%> (ø)` | :arrow_up: |
| #single | `40.74% <54.54%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.66% <93.18%> (-0.11%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.04% <0%> (-0.48%)` | :arrow_down: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `96.47% <0%> (-0.45%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.34% <0%> (-0.06%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.25% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=footer). Last update [38f41e6...d3d9f45](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=h1) Report
> Merging [#18496](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d74ac70ffd44e9d00e6ffa8ecf7a1a88312e8065?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18496/graphs/tree.svg?height=150&token=eZ4WkYLtcO&src=pr&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18496      +/-   ##
==========================================
- Coverage   91.44%   91.43%   -0.02%     
==========================================
  Files         157      157              
  Lines       51379    51374       -5     
==========================================
- Hits        46985    46972      -13     
- Misses       4394     4402       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.3% <100%> (ø)` | :arrow_up: |
| #single | `40.62% <72.22%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.69% <ø> (-0.03%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.46% <ø> (-0.01%)` | :arrow_down: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `97.11% <ø> (+0.19%)` | :arrow_up: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `96.54% <100%> (-0.01%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.8% <100%> (+0.04%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18496/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.81% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=footer). Last update [d74ac70...1582c42](https://codecov.io/gh/pandas-dev/pandas/pull/18496?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jorisvandenbossche @jreback Bringing here a conversation from Gitter: one possible way to approach this (avoiding the problem that ``{np.nan : 2}[float('nan')]`` raises ``KeyError``) is to modify ``lib.fast_multiget`` so that it iterates the dict and then matches to specific keys, rather than directly looking for such keys. Or it could iterate the dict only if ``np.nan`` appears in the desired keys (or if the desired keys are unspecified), and follow the usual behavior otherwise.
@toobaz happy to have you modify cython code directly to avoid perf penalty (and it also abstracts away some of the complexity). inline comments a plus!
@toobaz sidenote: can you give the PR a bit more descriptive title?
Since I introduced no (explicit) loops I preferred to keep code in ``*.py``.

I _could_ add a test at the beginning of ``lib.fast_multiget`` asserting the absence of NaN in the argument. However the case of tuples would be tricky: it's just more complicated - and not particularly useful I guess - to check if they contain NaNs, but at the same time raising an error for any tuple would be wrong.
Suggestion (but didn't think it fully through): we can make a distinction on whether `index` was passed or not. If `index` was passed, we do want to do dict-lookup (as otherwise it could potentially give a performance bottleneck) -> current code. If index is not passed, we can just get the keys, values of the full dict (as we want all those anyway, so no need to then do a dict-lookup for each key).

That doesn't solve the problem with `pd.Series({np.nan: 1, 2:2}, index=[1, np.nan,])`, but would already solve the majority of the cases I think?
> If index was passed, we do want to do dict-lookup (as otherwise it could potentially give a performance bottleneck) -> current code. If index is not passed, we can just get the keys, values of the full dict (as we want all those anyway, so no need to then do a dict-lookup for each key).

In my current version the lookup is not a bottleneck because it is just an iteration on keys/values, precisely what is needed anyway in the "no index" case (where, by the way, no reindexing happens because the indexes are ``identical()``). However, still in the "no index" case, keys are being iterated twice, and I can fix this at the cost of few lines in ``Series.__init__``.
@jreback ping
thanks!