COMPAT: np 1.18 wants explicit dtype=object)
State: closed

xref #30030, xref https://github.com/numpy/numpy/issues/15041, https://github.com/numpy/numpy/issues/15045

ATM we have 85 tests in the npdev build failing, this gets it down to 14 (at least for me locally).  

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I think we (cc @mattip) might not have communicated how to fix this problem well enough in NEP 34. As I see it, there are two cases when dealing with coercing object arrays:

* You know exactly what shape or dimension you want the output to be, so pre-allocate with `np.empty`, then fill:
    ```
    def as_1d(x):
        arr = np.empty(len(x), dtype=object)
        arr[:] = x
        return arr
    ```
    Now admittedly, it's not very easy to write such a function for anything other than 0d and 1d. Maybe we need to solve that before we undo the reversion.
* You don't know what shape the array is supposed to be. The caller is responsible for specifying this clearly, by performing the ambiguous conversion for you. Silencing the warning defeats the point of the deprecation:
    ```
    def do_some_things(arr, obj):
        arr = np.asarray(arr)   # if this emits a warning, it is the callers fault, not yours
    ```
    It's unfortunate that python doesn't make it easy to append context to a warning and propagate it, so you can explain to pandas users that _they_ need to fix the warning.
@jbrockmendel can you rebase
NumPy is considering adding something like `np.array(vals, dtype=np.legacy_behaviour)` so libraries (like pandas) could use that to keep the old behaviour. The idea would be that NumPy could add an additional API as mentioned in the [Alternatives](https://numpy.org/neps/nep-0034.html#alternatives) section in the NEP, or libraries could work out strategies that avoid using NumPy for this kind of container, and until then you could continue using the old behaviour.
> NumPy is considering adding something like `np.array(vals, dtype=np.legacy_behaviour)` so libraries (like pandas) could use that to keep the old behaviour. The idea would be that NumPy could add an additional API as mentioned in the [Alternatives](https://numpy.org/neps/nep-0034.html#alternatives) section in the NEP, or libraries could work out strategies that avoid using NumPy for this kind of container, and until then you could continue using the old behaviour.

thanks @mattip 

yeah we likely need a combination of strategies here; e.g. to fix where we can be sure of what the input is and suppress / remove where needed.
@jbrockmendel can you rebase again
needs a rebase again
Just to make sure you guys are up to date. It is very likely that the Opt-In to the old behaviour will not be `object`, but rather a new sentinel object.
> Just to make sure you guys are up to date. It is very likely that the Opt-In to the old behaviour will not be object, but rather a new sentinel object.

There is a difference between `np.array([1, 2], dtype=object)` and `np.array([1, 2], dtype=np.sentinel_that_means_legacy_behaviour)`: the first will result in `dtype=object`, the second in `dtype=int` See numpy/numpy#15083
> Just to make sure you guys are up to date. It is very likely that the Opt-In to the old behaviour will not be object, but rather a new sentinel object.

@seberg @mattip thanks.  Is there a way to re-enable the warning so I can see where we need to update things?

You could build numpy with numpy/numpy#15119. I just pushed it so you might want to wait to see if it passes all tests before trying it out.
Closing, will work on this locally.
@jbrockmendel did you already make progress on this? I am curious if an `np.allow_object` flag to opt-in to the old behaviour solves the problem (also, assuming that we may want to remove it eventually). E.g. I could imagine that it may be convenient to be able to opt-in to the future error immediately.

In the best case, the cases that are not easily fixed are cases where `np.allow_object` can be pushed to the end-user as a (potentially) intermediate solution.
> did you already make progress on this? I

We've updated most of the places that were easy to just add an explicit "dtype=object".

I haven't gotten around to work through this locally as suggested [here](https://github.com/pandas-dev/pandas/pull/30035#issuecomment-566208794).  Do you agree that would be a good next step (and if so, is that still the right branch to use?)?
Good question, mostly good to know for us right now. If you (and we) are fine with `dtype=object` allowing ragged object arrays and for you using `object` indiscriminately is actually OK we are fine (we may have to think about it again at NumPy). In most cases that can run into this (and the warning/error is incorrect), I would hope you can even do `np.empty(..., dtype=object)` and set the shape manually.
We need to figure out what downstream requires here to know what our realistic options are.

> and for you using object indiscriminately is actually OK

No, there are still many places where we currently call `np.array` without passing a dtype and depend on numpy's type inference.  In many/most of those cases, we won't have pre-checked for non-raggedness.

There are a good chunk of those cases where we know we want 1D output, so as soon as we see a listlike element we know it'll end up object-dtype.  Would passing `ndim=1` in some cases where we can't specify dtype make this any easier?
@jbrockmendel adding an `ndim` keyword argument is one of the options yes. We would prefer to add only the things that are really needed though. We could also add the opposite (do not allow ragged) although that forbids using the `allow_object` name for other things in the future. In which case your pattern would be to use a `try/except`. The `try/except` idea seems only really useful though if: 1. You want to deprecate this also and 2. not doing anything (forcing the user to fix it up) is not an option.

Do you have a start with the particularly tough spots? Maybe we have to look at it from numpy to get a better feel for the pain in pandas. I would be happy if pandas is just fine with `np.allow_object/ragged`, but even then it would be nice if at some far future we could move away from supporting it entirely.
> Do you have a start with the particularly tough spots?

I just  cloned [this](https://github.com/numpy/numpy/pull/15119) branch and will test against it shortly.
i didnt get any of the expected errors on that branch.  is there another one i should try?
xref numpy/numpy#15119