Repository: pandas-dev/pandas
Pull request #60661 (2025-01-05 14:10:09)

String dtype: disallow specifying the 'str' dtype with storage in [..] in string alias
The intention was for the new default `"str"` dtype to not include the storage in the string alias, and so to also not allow constructing it that way (this is discussed in the PDEP).

This is also implemented this way, as you can see when directly calling the extension dtype API:

```python
# for "string" this is allowed
>>> pd.StringDtype.construct_from_string("string[pyarrow]")
string[pyarrow]

# but for "str" not
>>> pd.StringDtype.construct_from_string("str[pyarrow]")
...
TypeError: Cannot construct a 'StringDtype' from 'str[pyarrow]'
```

However, when specifying this as a `dtype` argument in eg constructors (going through `pandas_dtype(...)`, which goes through the extension dtype registry), this "accidentally" kind of works, but gives an unexpected result:

```python
# this raises as expected
>>> pd.Series(["a"], dtype="str[python]")
TypeError: data type 'str[python]' not understood

# but this gives a result
>>> pd.Series(["a"], dtype="str[pyarrow]")
>>>
0    a
dtype: string[pyarrow]
>>> type(pd.Series(["a"], dtype="str[pyarrow]").dtype)
pandas.ArrowDtype
```

I think it is confusing that it does work in case of the pyarrow storage, but then does give a different dtype than what you would typically expect. 
So I would rather just disallow this case (which is what this PR does), although this is a small breaking change for people currently using `dtype="str[pyarrow]"` to get the ArrowDtype.

rhshadrach: > I think it is confusing that it does work in case of the pyarrow storage, but then does give a different dtype than what you would typically expect. So I would rather just disallow this case (which is what this PR does), although this is a small breaking change for people currently using `dtype="str[pyarrow]"` to get the ArrowDtype.

This gives a definitively wrong result and so I think that puts it solidly in the bugfix camp, not a breaking change. 


WillAyd: If we really cared we could convert that during the pickle read, although I dont think that would be a blocker. Generally using pickle to move from one environment to another is discouraged
jorisvandenbossche: > If we really cared we could convert that during the pickle read, although I dont think that would be a blocker. Generally using pickle to move from one environment to another is discouraged

@WillAyd that comment about pickle was meant for the discussion in https://github.com/pandas-dev/pandas/issues/60639?

Patch:
diff --git a/pandas/core/dtypes/dtypes.py b/pandas/core/dtypes/dtypes.py
index 1dd1b12d6a..1eb1a63005 100644
--- a/pandas/core/dtypes/dtypes.py
+++ b/pandas/core/dtypes/dtypes.py
@@ -2339,7 +2339,7 @@ class ArrowDtype(StorageExtensionDtype):
             )
         if not string.endswith("[pyarrow]"):
             raise TypeError(f"'{string}' must end with '[pyarrow]'")
-        if string == "string[pyarrow]":
+        if string in ("string[pyarrow]", "str[pyarrow]"):
             # Ensure Registry.find skips ArrowDtype to use StringDtype instead
             raise TypeError("string[pyarrow] should be constructed by StringDtype")
         if pa_version_under10p1:


Test patch:
diff --git a/pandas/tests/dtypes/test_common.py b/pandas/tests/dtypes/test_common.py
index 5a59617ce5..fa48393dd1 100644
--- a/pandas/tests/dtypes/test_common.py
+++ b/pandas/tests/dtypes/test_common.py
@@ -837,6 +837,26 @@ def test_pandas_dtype_string_dtypes(string_storage):
     assert result == pd.StringDtype(string_storage, na_value=pd.NA)
 
 
+def test_pandas_dtype_string_dtype_alias_with_storage():
+    with pytest.raises(TypeError, match="not understood"):
+        pandas_dtype("str[python]")
+
+    with pytest.raises(TypeError, match="not understood"):
+        pandas_dtype("str[pyarrow]")
+
+    result = pandas_dtype("string[python]")
+    assert result == pd.StringDtype("python", na_value=pd.NA)
+
+    if HAS_PYARROW:
+        result = pandas_dtype("string[pyarrow]")
+        assert result == pd.StringDtype("pyarrow", na_value=pd.NA)
+    else:
+        with pytest.raises(
+            ImportError, match="required for PyArrow backed StringArray"
+        ):
+            pandas_dtype("string[pyarrow]")
+
+
 @td.skip_if_installed("pyarrow")
 def test_construct_from_string_without_pyarrow_installed():
     # GH 57928
diff --git a/pandas/tests/strings/test_get_dummies.py b/pandas/tests/strings/test_get_dummies.py
index 3b989e284c..541b0ea150 100644
--- a/pandas/tests/strings/test_get_dummies.py
+++ b/pandas/tests/strings/test_get_dummies.py
@@ -6,6 +6,7 @@ from pandas._config import using_string_dtype
 import pandas.util._test_decorators as td
 
 from pandas import (
+    ArrowDtype,
     DataFrame,
     Index,
     MultiIndex,
@@ -113,8 +114,10 @@ def test_get_dummies_with_str_dtype(any_string_dtype):
 # GH#47872
 @td.skip_if_no("pyarrow")
 def test_get_dummies_with_pa_str_dtype(any_string_dtype):
+    import pyarrow as pa
+
     s = Series(["a|b", "a|c", np.nan], dtype=any_string_dtype)
-    result = s.str.get_dummies("|", dtype="str[pyarrow]")
+    result = s.str.get_dummies("|", dtype=ArrowDtype(pa.string()))
     expected = DataFrame(
         [
             ["true", "true", "false"],
@@ -122,6 +125,6 @@ def test_get_dummies_with_pa_str_dtype(any_string_dtype):
             ["false", "false", "false"],
         ],
         columns=list("abc"),
-        dtype="str[pyarrow]",
+        dtype=ArrowDtype(pa.string()),
     )
     tm.assert_frame_equal(result, expected)