ENH: Add dtype argument to str.decode
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Ref: https://github.com/pandas-dev/pandas/pull/60795#discussion_r1947937837

PyArrow-backed strings cannot handle surrogates. When users have `infer_string=True` and PyArrow installed, they can end up with a failure they can't workaround when calling `str.decode`. Adding the dtype argument allows for a workaround.

Labels:
name: Enhancement
name: Strings
description: String extension data type and string data

Comments:
> Maybe add a test with actual surrogates to make sure the `dtype=object` workaround works in that case?

There is a surrogate in `test_decode_object_dtype`.
Have to think about this some more but my initial reaction is a -1. If someone needs non UTF-8 support I think we should push them to using the object dtype, rather than extending the API like this
> > Maybe add a test with actual surrogates to make sure the `dtype=object` workaround works in that case?
> 
> There is a surrogate in `test_decode_object_dtype`.

Sorry, missed that!



> If someone needs non UTF-8 support I think we should push them to using the object dtype, rather than extending the API like this

That's exactly what this PR is intending to enable? With this keyword, people can choose to use object dtype explicitly (without it, we always use str dtype, which then fails)
> With this keyword, people can choose to use object dtype explicitly (without it, we always use str dtype, which then fails)

Ah sorry I am getting tripped up over terminology and the existing API. I'm guessing this is a Python2 relic that we even offer `str.decode`, since there is no string method in Python3 for decode. 

So makes sense, I just want to really limit the use of keywords to control data types because as they can be hard to reason about. I don't think it can be avoided specifically for this method though
Thanks @rhshadrach 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 63249f2aa95ef0b0300ea2f1cc68200cc8b13484
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60940: ENH: Add dtype argument to str.decode'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60940-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60940 on branch 2.3.x (ENH: Add dtype argument to str.decode)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
> I'm guessing this is a Python2 relic that we even offer `str.decode`, since there is no string method in Python3 for decode.

It's [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode), I think we just don't want to add a `bytes` namespace so we stashed it under `str`.
Back in Python2 the str type had a decode method, as it was closer to an array of bytes than the unicode object type that it currently is in Python3. The accessor being under that namespace today is a bit of cruft owing back to that legacy.

Certainly not something to fix here, but its an interesting API nonetheless
Actually giving this a little more thought, instead of adding a `dtype` argument what if we coerce to object when the encoding is anything but `utf-8` and just use the standard string inference logic when it is `utf-8`?
I'm negative on adding values-specific behavior.
Backported in https://github.com/pandas-dev/pandas/pull/60968