ENH: Allow `s.map(d, na_action='raise')`
State: closed

Addresses #14210

Often users use `s.map()` blindly (without checking the original series beforehand due to their expectations) and are surprised by downstream errors which are caused by the implicit replacement of values to `np.nan` if they are not part of the mapping dictionary:

```python
import pandas as pd

s = pd.Series(["cat", "dog", "mouse"])
d = {"cat": "kitten", "dog": "puppy"}

res = s.map(d)

print(res)
# 0    kitten
# 1     puppy
# 2       NaN
# dtype: object
```

This example [here](https://github.com/pandas-dev/pandas/issues/14210#issuecomment-246684563) also shows how a typo in the input data can cause unwanted trouble.

This PR adds a possibility to raise an error via `s.map(d, na_option='raise')`, if the mapping does not cover all values in the array.

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Still interested!
Sorry for pinging @mroeschke - can you please spend a quick glimpse on that, if the design and the goal is worth following ... thank you in advance!
Sorry @kopytjuk nobody checked this earlier. While I'm ok to find a solution to the use case being addressed here (raise when the dict in map doesn't contain all the values in the series/dataframe), I'm -1 in the approach here.

The main reason is that I think we should get rid of the `na_action` parameter. Which I find cumbersome, and reading the issue descriptions confirms that not even very experienced pandas core devs understand what the parameter does without proper research.

Second, `na_action` refers to the action to so when missing values are found in the data. So, `na_action="ignore"` means that missing values should be ignored and not passed to the function. If things are logical and consistent, to me `na_action="raise"` would mean that if missing values are found in the original data, we would raise. Which is different to what's being implemented here. Adding even more confusion to an already confusing API.

I'm personally not a big fan of adding a new parameter to `.map`. In particular because `.map` is already two functions in one, mapping with a dictionary, and applying a function elementwise, and the new parameter would be tricky when considered for both cases. But still this would be a better option than using `na_action`.

I think the best solution here is that you have a validation after the map to ensure not missing values are there. I don't think there is a pure pandas one-liner to do it, and maybe adding it is not a bad idea. But I think there are third party libraries to ensure data consistency in pandas, maybe that's the way to go.

What do you think?
Thanks for your reply @datapythonista!

> If things are logical and consistent, to me na_action="raise" would mean that if missing values are found in the original data, we would raise. Which is different to what's being implemented here.

I agree on that, the "na_action" is not a fitting argument name for what I implemented, since the argument relates to the original (series) data and not the keys of the mapping.

> I think the best solution here is that you have a validation after the map to ensure not missing values are there. I don't think there is a pure pandas one-liner to do it, and maybe adding it is not a bad idea. But I think there are third party libraries to ensure data consistency in pandas, maybe that's the way to go.

I agree on that as well, libraries like `pandera` are often used in data engineering use-cases to ensure that the resulting dataframe (after a chain of operations) follows a certain schema (column names, datatypes and existence of NaNs).

In addition to your points you made: a novice user won't activate the proposed `na_action="raise"` argument anyway, which he does not understand (as you stated, the understanding of the mechanics `na_action` is not straighforward) ...

So in summary, after weeks of distance to my proposal, I agree - I am also "-1" on the `na_action="raise"` approach.

Do you think, raising a warning would be a good alternative approach? Especially in explorative contexts (in which a developer looks at the data in a notebook) such a warning would be helpful, to make sure that the user does not miss a value in the mapping. I think warnings help to develop a more stable code, because people start looking into their data more deeply and take a look at their code, especially if parts of it are LLM generated.

What do you guys think about the "warning-only-without-any-additional-arguments" approach in case the mapping does not cover the original data?

This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Sorry I forgot to answer @kopytjuk. Regarding adding the warning, I think it'd be annoying for the users who are ok with the mapping not changing all the elements. I think it'd be nice to better address the case you are considering from the pandas side, but I don't really see how. Feels like while not ideal, the best is for users to address the problem themselves, asserting that the series or dataframe only have values in the mapping before the map, checking for NA after the map, or using a function instead of a dict, that raises when an element is not in the mapping.

I'll close this PR, as it's not the approach we want to do. Feel free to continue the discussion in the issue, or open a new one more specific to this problem.
I understand the logic here -- that `na_action` is the wrong parameter -- but I definitely think a one-liner for "map using this dictionary, error if the value isn't present" _should_ be in pandas. In fact it would have been better if that had been the default, because the user could have explicitly used a `defaultdict` when they didn't want errors, which is still easy to do inline.

Unfortunately that ship has sailed and I know there's no way the default would be changed now. But I'm +1 on adding a new parameter to allow this behavior. I hear the argument that `.map` is already complicated, but silently introducing nulls like this is a big mistake IMO. Just like how I use `dropna=False` on every `value_counts` call because that should have been the default, a new parameter would quickly become muscle-memory for me every time I use `.map` with a dictionary.
Any thoughts @kopytjuk or @datapythonista?
`na_action` should be deprecated in #61530, and `skipna` is introduced instead. In an ideal world we would have separate methods for `map` with a `dict` and `map` with a function, since the parameter you describe won't make sense for functions. But there are worse things in the pandas API, I'm personally not against a new boolean parameter to raise if a value is not found in the dictionary. And even having `True` as the default, but that's trickier since it's a breaking change, and not sure what how exactly the transition path would look like.
Hello guys, sorry for the late reply! Since I introduced the idea enhancement, I am still in favor of it 👍 

Transition-wise, I still think even if we introduce an additional flag and keep the current behavior unchanged, imo it would be still a helpful flag in exploratory settings. At a later stage we can think of how to transition the defaults.

Since it would be a new flag, I would suggest a flag `not_in_map_action: Literal[None, "forward", "raise"] = None` with the current behavior mapping the value to `NaN` as the default. In case of `forward` we could forward the original element, in case of `raise` we raise an exception (e.g. `KeyError`).

I would implement and maintain the required changes if the core maintainers @datapythonista @mroeschke etc. agree on the way forward.

P.S. Maybe one [glimpse](https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.replace_strict.html) to the neighbors at polars how they deal with this issue we are discussing: they define a `replace_strict` function and raise an Exception, in case a default value is not provided.