Remove use_nullable_dtypes and add dtype_backend keyword
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@mroeschke this is heavy unfortunately. Should be ready for a first look for wording and similar, I expect tests to fail since http tests timed out locally which made running tests tricky. Will fix when ci is through

I think we could split theoretically, 

- sql
- arrow stuff (orc, parquet, feather)
- to_numeric and convert_dtypes
- everything else

Let me know what you prefer

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Arrow
description: pyarrow functionality

Comments:
Looks good in general. I do think it'd make things much easier to split this in multiple PRs. In particular for you, and we could work in the different components in parallel, so you don't need to do all the work yourself. In any case, great that you already have this, and since it looks like the CI is almost green here, also ok to merge all together, makes reviewing more complex than by parts, but fine with me.

Only to general questions:

- Why was it decided to use nodefault instead of "numpy" for the numpy dtypes? Seeing the PR seems to make things more complex and difficult to understand than using `dtype_backend='numpy'` by default.

- Why in some places we're using `use_backend_dtype: bool`? Again, seems like propagating `dtype_backend` and checking if different from numpy would be clearer (if that's what it means to use dtype backend, which is not very clear)

I guess I'm missing something, I don't see the advantage.
Answers to your two questions:

We agreed that no one should set numpy explicitly as dtype_backend right now, so we don’t allow it. I’ll create a follow up pr raising on invalid inputs after this is in.

i am using use_dtype_backend in situations where there is a second condition involved (like in read_csv where a supplied dtype takes precedence over the dtype_backend) or when we’d have to check many times for the default, this made it less readable to me
> We agreed that no one should set numpy explicitly as dtype_backend right now

I see. Is this with the idea to eventually deprecate the numpy backend? Or why don't we want users to be able to use `dtype='numpy'`?

> i am using use_dtype_backend in situations where there is a second condition involved (like in read_csv where a supplied dtype takes precedence over the dtype_backend) or when we’d have to check many times for the default, this made it less readable to me

Makes sense, thanks for clarifying, it wasn't obvious from the diff.

> I see. Is this with the idea to eventually deprecate the numpy backend? Or why don't we want users to be able to use dtype='numpy'?

I think the rational is to setup a future where we can more easily change the default dtype backend to `numpy_nullable` or `pyarrow`.
> I think the rational is to setup a future where we can more easily change the default dtype backend to `numpy_nullable` or `pyarrow`.

@mroeschke @phofl  I'm still a bit confused, sorry. If we allow `dtype_backend='numpy` then users can use that, and make sure that a future change in the pandas default won't change their backend. I fail to see how removing the `numpy` backend option is helping.


@jorisvandenbossche mind chiming in with your original reason for not having a `dtype_backend="numpy"` option (we didn't document this sufficiently in our meeting notes)
How would this interact with the ``dtype`` keyword for functions like ``read_csv``?

Should that overwrite dtype backend or should the pyarrow equivalent be returned?
e.g.
```
>>> import pandas as pd
>>> pd.options.mode.nullable_dtypes = True
>>> pd.options.mode.dtype_backend = "pyarrow"
>>> from io import StringIO
>>> pd.read_csv(StringIO("a,b,c\n1,2,3"), dtype="int64")
```
Explicit dtypes take precedence 
My bad, did not read comment above.
This is also the most logical thing. A possible use case: users want numpy dtypes except string columns, they should be arrow backed for example. There are probably more where Mixing to a certain extent is beneficial 
Getting back to numpy backend: right now a numpy backend doesn’t make any sense, it’s the default. Nobody should have to request it explicitly. In a future where we potentially switch the backend option (not sure if/when we get there) we have to support the new default backend everywhere. This would also mean that we would need a global option to opt into, this is where a numpy backend could make sense, depending on what we want to do exactly. At this point we can either get rid of the keyword or add a new option (probably the first). So right now it doesn’t help anyone and also doesn’t block anything in the future 
> So right now it doesn’t help anyone and also doesn’t block anything in the future

Yes, that sums it up for me. There is really no reason that anyone should now explicitly do `dtype_backend="numpy"`, so why would we then allow it? 
Not having it keeps it more flexible to do whatever we want later (we can still add it later if at some point that makes sense).

(one could also argue about the name "numpy" being slightly confusing/incorrect, since they are not all numpy dtypes (categorical, datetimetz, interval, etc), although they all use "numpy semantics", one could say) 

Merging this to rebase the other pr. Happy to address comments regarding wording in a follow up
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 ab76540ada429ffb6f10785f1623c47a997b5763
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #51853: Remove use_nullable_dtypes and add dtype_backend keyword'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-51853-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #51853 on branch 2.0.x (Remove use_nullable_dtypes and add dtype_backend keyword)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).