API: consistent NaN treatment for pyarrow dtypes
State: open

This is the third of several POCs stemming from the discussion in https://github.com/pandas-dev/pandas/issues/61618 (see #61708, #61716). The main goal is to see how invasive it would be.

Specifically, this changes the behavior of pyarrow floating dtypes to treat NaN as distinct from NA in the constructors and `__setitem__` (xref #32265).  Also in to_numpy, .values

Notes:
- [x] 
This makes the decision to treat NaNs as close-enough to NA when a user explicitly asks for a pyarrow integer dtype.  I think this is the right API, but won't check the box until there's a concensus.
 Changed this following Matt's opinion.
- [x] I still have 
113
 
89
 
9
 0 failing tests locally.  
Most of these are in json, sql, or test_EA_types (which is about csv round-tripping).

- [x] Finding the mask to pass to pa.array needs optimization.
- [x] 
The kludge in NDFrame.where is ugly and fragile.
 Fixed.
- [ ] Need to double-check the new expected in the rank test.  Maybe re-write the test with NA instead of NaN?
- [x] Do we change to_numpy() behavior to _not_ convert NAs to NaNs? this would be needed to make test_setitem_frame_2d_values tests pass

Labels:
name: PDEP missing values
description: Issues that would be addressed by the Ice Cream Agreement from the Aug 2023 sprint

Comments:
@mroeschke when convenient id like to get your thoughts before getting this working.  it looks pretty feasible.
Generally +1 in this direction. Glad to see the changes to make this work are fairly minimal
Not able to judge the implementation, but I'm +1 on the concept.
While I am personally in favor of distinguishing NaN and NA, I think most of the changes here involve distinguishing NaN when constructing the arrays? (so eg constructing the pyarro-based EA from user input like numpy arrays?) 

Personally, I think that is a change we should only make _after_ making those dtypes the default, and probably even years after that after a very long deprecation process. 
(currently _everyone_ who is creating pandas DataFrames from numpy data assumes that the NaNs in the numpy data is considered as missing. IMO that is a behaviour that we will have to keep (for a long time) even if we distinguish NaN and NA)

> I think most of the changes here involve distinguishing NaN when constructing the arrays?

Yes.  Constructors (which affect read_csv) and `__setitem__` are most of this.

> I think that is a change we should only make after making those dtypes the default, and probably even years after that after a very long deprecation process.

My current thought (will bring up on today's dev call) is that we should add a global flag to enable both never-distinguish (see #61708) as the default and always distinguish (this) as opt-in.
Based on last week's dev call, I am adapting this and #61708 from POCs to real PRs.  This implements a global flag `"mode.nan_is_na"` (default `True`) to choose which behavior we want.

This PR only implements this for ArrowEA.  #61708 will do the same for the numpy-nullables.  (I have a branch trying to do it all at once and it is getting ungainly).  A third PR will add tests for the various issues this closes.