DOC: clarify pitfalls of NA vs NaN in nullable floats
State: closed

- [x] stopgap solution addressing  #49818 and such until the discussion in #32265 is resolved 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Following up on my [comment](https://github.com/pandas-dev/pandas/issues/32265#issuecomment-1405543022): since the current behavior easily leads to mistakes when trying to detect missing/NaN data and does not match the docs' described semantics, this PR is trying to update the docs to clarify this edge case so users can avoid the pitfalls.

Notes/questions on PR work: 
* There may be other places to update, but I figure once the language is approved it's easy to add it elsewhere
* Should we link to the issue #32265 discussing what the behavior should be? That would enable users to weigh in with their preferences but might be too detailed/confusing.
* Should the docstring describe this scenario in terms of Arrays or the corresponding dtypes?

Thanks!


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Stale

Comments:
Asking @jorisvandenbossche if interested in reviewing, as suggested
Hi @jorisvandenbossche, I've tried to slim down the language to address your feedback - thanks.

However, I did realize a slight issue - I wanted to give guidance on using both `Float<>` and `float<>[pyarrow]` dtypes, since pandas is starting to support both as "nullable dtype backends". But while users can detect NaN with `np.isnan` for the former, it seems like there isn't a _documented_ way for the latter (pyarrow has `pc.is_nan/is_null` but getting the underlying arrow array seems to require private attributes). Wanted to bring that up in case it makes sense to have pandas offer something more there. For this PR, I guess we can just only cover the Float case?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

I am still interested in completing this! I think it needs a little bit of pandas maintainer feedback - at least, to clarify whether there’s a supported way to detect NaN values in an arrow-dtype array.
I think the topic of https://github.com/pandas-dev/pandas/issues/32265 will be discussed and decided on soon which should address this so closing for now
> I think the topic of #32265 will be discussed and decided on soon which should address this so closing for now

@mroeschke 10 months later it is still being decided. How about re-opening this PR?

I am under impression that some sort of consensus has started to crystallise around options [2](https://github.com/pandas-dev/pandas/issues/32265#issuecomment-1651989409), [3](https://github.com/pandas-dev/pandas/issues/32265#issuecomment-1670110335), or [4](https://github.com/pandas-dev/pandas/issues/32265#issuecomment-1678324585) with the inclination towards the current behaviour (`s.isna()` returns `False` for `NaN`). 

I propose to also add a clarification to "User Guide". Nothing in the documentation says explicitly that `NaN` is no longer treated as missing in nullable float arrays. On the contrary, [examples with `df.fillna(0)`, `df.ffill()` and `df.bfill()`](https://pandas.pydata.org/docs/dev/user_guide/missing_data.html#filling-missing-data) clearly suggest that `NaN` is equivalent to `NA` in `float64[pyarrow]`.

Placing the link to the aforementioned discussion in documentation seems like a good idea to me as a user.