TST[string]: update expecteds for using_string_dtype to fix xfails
State: closed

It isn't 100% obvious that the new repr for Categoricals is an improvement, but it's non-crazy.  One of the remaining xfails one is for `eval(repr(categorical_index))` round-tripping that won't be fixable unless we revert back to the old repr behavior.

I'm pretty sure that the fix in test_astype_dt64_to_string is correct and the test is just wrong, but merits a close look.

That leaves 12 xfails, including the one un-fixable round-trip one that we'll just remove.  Of those...
- [x] test_join_on_key i think is surfacing an unrelated bug that I'll take a look at (xref #61771)
- [x] test_to_dict_of_blocks_item_cache is failing because we don't make series.values read-only for ArrowStringArray.  I think @mroeschke can comment on how viable/important that is.
- [ ] test_string_categorical_index_repr is about CategoricalIndex repr that span multiple lines; with the StringDtype the padding is changed.
- [x] 4 in pandas/tests/io/json/test_pandas.py that im hoping @WillAyd can take point on 
- [ ] test_to_string_index_with_nan theres a MultiIndex level that reprs with a `nan` instead of `NaN`.  Not a huge deal but having mixed-and-matched nans/NaNs in the repr is weird.
- [ ] test_from_records_sequencelike i don't have a good read on
- [x] tests.base.test_misc::test_memory_usage is skipped instead of xfailed, but the reason says that it "doesn't work properly" for arrow strings which seems xfail-adjacent.  Instead of skipping can we update the expected behavior cc @jorisvandenbossche ?

(Update: looks like I missed one in test_http_headers and another in test_fsspec)

Labels:

Comments:
The JSON issues stem back to the fact that:

```python
>>> pd.Series([None, '', 'c']).astype(object)
```

yields different behavior with/without the future string dtype. In the "old" world, this would preserve the value of `None` but in the new world, `None` gets cast to a missing value indicator when contained within a series of string values.

In theory we could try and work around those semantics by natively supporting an object type in the JSON reader, but that's a ton of effort and I don't think worth it, given JSON does not natively support object storage
thanks, will update those tests' `expected`s
> we don't make series.values read-only for ArrowStringArray

Can't speak for the viability but I think this _should_ be read-only per CoW. Is this related to the underlying `ArrowExtensionArray.__setitem__` immutably issue?
> Can't speak for the viability but I think this should be read-only per CoW. Is this related to the underlying ArrowExtensionArray.__setitem__ immutably issue?

I suspect we would need a mechanism like ndarray.flags for EAs to declare an object as read-only.  Definitely out of scope for this PR.

Looking at the test, the pertinent behavior can be tested by making the column specifically object dtype.
> * tests.base.test_misc::test_memory_usage is skipped instead of xfailed, but the reason says that it "doesn't work properly" for arrow strings which seems xfail-adjacent.  Instead of skipping can we update the expected behavior cc @jorisvandenbossche ?

I quickly checked that one, and I am not entirely sure why we skipped it based on "doesn't work properly for arrow strings". The issue with the test seems to be that it is assuming too much about the input data, and its `is_object` definition is no longer correct. For example, a MultiIndex gets considered as object dtype, but now if all levels of the MultiIndex are using `str` dtype, then the test should not actually see it as `is_object`. 
Same for `is_categorical` where the test code assumes that a categorical is using object dtype categories, I think.

(can do a separate PR to fix this one, now that I already dived into it)
Looks like #61757 wasn't backported, so i dont think this one should be either
> > > we don't make series.values read-only for ArrowStringArray
> > 
> > Can't speak for the viability but I think this _should_ be read-only per CoW. Is this related to the underlying `ArrowExtensionArray.__setitem__` immutably issue?
>
> I suspect we would need a mechanism like ndarray.flags for EAs to declare an object as read-only. Definitely out of scope for this PR.

That was actually one of the remaining TODO items for the CoW implementation, ot indeed add a similar readonly flag mechanism to EAs to declare them as read-only. 

Experimenting with that in https://github.com/pandas-dev/pandas/pull/61925