PDEP-18: Nullable Object Dtype
State: open

as per PDEP-1 

> The initial status of a PDEP will be Status: Draft. This will be changed to Status: Under discussion by the author(s), when they are ready to proceed with the decision making process.

but comments are surely welcome in the meantime.

Labels:
name: Stale
name: PDEP
description: pandas enhancement proposal

Comments:
“Object” analogous to “Float64”?
At least to me the PDEP will be easier to read (and comment on) if you limit the line width, to 80 or similar.

The idea sounds good, it'd be good if you can provide information on how using a boolean mask compares to having `pandas.NA` inside the main array.
> At least to me the PDEP will be easier to read (and comment on) if you limit the line width, to 80 or similar.

sure.

> it'd be good if you can provide information on how using a boolean mask compares to having `pandas.NA` inside the main array.

using a sentinel as opposed to a mask is an implementation detail that I can expand on. I'm assuming that this would still be a separate dtype from the traditional numpy dtype, a pandas nullable dtype? We have the string array backed by a masked object array that I was effectively proposing reusing/refactoring as a base class.

There is also another option which is maybe what you are proposing: making a breaking change to the exisiting numpy object dtype to handle pd.NA differently? This is perhaps what is in the rejected ideas section and that needs clarification?
> “Object” analogous to “Float64”?

That's the obvious choice but IIRC the capitalization was considered as confusing/non intuitive by some when discussed with respect to the string dtype.

> > “Object” analogous to “Float64”?
> 
> That's the obvious choice but IIRC the capitalization was considered as confusing/non intuitive by some when discussed with respect to the string dtype.

naming the nullable object dtype "Object" aligns well with pandas’ approach to evolving its dtypes (like "Int64" for integers, "Float64" for floats and "Boolean" for the nullable `bool` dtype). It creates a clear visual and semantic cue for users that a nullable, extension-based implementation is being used. As long as the design and documentation explicitly address the differences between the legacy object dtype and "Object", this approach could indeed enhance clarity and usability even though it it less explicit that "object_nullable".

Also bear in mind that, being effectively a tweek to the pd.NA variant of the python backed string dtype, the repr could be just "object", just as "string[pyarrow]" is shown as just "string". This effectively indicates the logical type and that instead of using the dtype string alias we could recommend constructing the nullable array for testing/evaluation using the Dtype object using the patterns that there seemed to be some consensus on in PDEP-13. There may be advantage of using "object" as the repr instead of "Object" as this could potentially simplify the transition to nullable types by default in future. So then I would think that using the more explicit "object_nullable" for now could maybe be better than introducing the capitalized form if it was agreed that the repr is just "object" without the subtleties of capitalization.

Also note that in the PDEP it was written "tentatively named `"object_nullable"`" based on the passage of PDEP-14 which needed a sub discussion to address. The words chosen purposely not to set the dtype string alias in stone and allowing the discussion to potentially avoid this debate in the main discussion thread.
> > it'd be good if you can provide information on how using a boolean mask compares to having `pandas.NA` inside the main array.
> 
> using a sentinel as opposed to a mask is an implementation detail that I can expand on. I'm assuming that this would still be a separate dtype from the traditional numpy dtype, a pandas nullable dtype? We have the string array backed by a masked object array that I was effectively proposing reusing/refactoring as a base class.
> 
> There is also another option which is maybe what you are proposing: making a breaking change to the exisiting numpy object dtype to handle pd.NA differently? This is perhaps what is in the rejected ideas section and that needs clarification?

No matter which of the two approaches above is considered, I think the arguments for using a mask as opposed to a sentinel are probably the same:

Using a Boolean mask is generally seen as the preferred design in pandas for extension arrays. It provides uniform missing-value handling, clearer data semantics, and is more in line with how extension types like "Int64" and "boolean" have been developed, not to mention the nullable string array which it is intended to re-use for the nullable object implementation.

This separation is one reason why the design of a nullable object dtype would benefit from a dedicated missingness mask rather than trying to “magically” interpret pd.NA embedded in an otherwise generic Python object array. IIUC pd.NA was designed as the **representation** of missing values in nullable arrays and was never intended to be an explicit sentinel value. I think many of the issues with pd.NA in object arrays arise from users thinking that the pd.NA object itself is a missing value and not a representation of a missing value. Of course a python object array can hold any object and so we can't, or maybe don't, stop users putting the pd.NA object in the traditional numpy object array.

To compare the two approaches:

Embedded pd.NA:

- Maybe a simpler conceptual model for small or homogeneous arrays where overhead is minimal.
- Checking each element at runtime might involve extra comparisons
- When missing values are stored in the same array as valid data, ensuring that operations treat pd.NA consistently can be tricky. 

Separate Boolean Mask:

- Offers a clear and robust way to denote missing data while preserving clean data arrays; aligns with the design of other extension types in pandas; facilitates efficient and consistent missing data handling across operations
- Introduces additional complexity in data structure design. This is not really an issue as there is no POC needed as the nullable object array shares so much code with the tried and tested pd.NA variant of the string array, available for a long time now.
- Separating the missingness information from the actual data often leads to more readable, more performant and more maintainable code. When operations are performed, pandas can first consult the mask to identify missing elements and then process only the valid ones, or process only the missing values with operations such as `fillna`. In vectorized operations, many operations can short-circuit based on the mask.
- The extra Boolean mask does add memory overhead, though future implementations could optimize the mask as a bitmask rather than a full Boolean array. This is applicable to all pandas nullable types and so that discussion would be outside the scope of this PDEP.
 

A few API question it'd be helpful to see addressed explicitly:

1) `ser[0] = np.nan`  Does this assign NaN or does this silently replace with pd.NA?
2) Same question with NaN in a list passed to the constructor.
3) Same questions but with None or pd.NaT
4) `ser = pd.Series([pd.NA, None, pd.NaT, np.nan], dtype="object_nullable")`.  (Assuming we don't silently replace) What do isna/fillna/skipna do?

I think @jbrockmendel questions are very good and worth having explicit in the proposal.

Personally I think `pd.NA` should be used for setting values in the boolean mask, everything else should go into the values and not have special treatment.

While it may be counterintuitive at first based on past behavior, I think it's the simplest to implement, to explain, and to understand.
Just posting high level concerns:

1. While pandas still `object` a "string dtype", would `"object_nullable"` be yet-another-string-dtype with an alternative NA semantics? I would be more comfortable with this type if pandas just considers `object` as purely PyObjects
2. Maybe a meta comment about this topic, I do think discussing a "nullable object type" would be better suited in a PDEP that discusses all "nullable types" (system) to avoid fragmentation/diversion of terminology, null semantics, etc. as inevitability the other types will be discussed as well.
Sorry if I miss it, but is the plan to implement the nullable object backed by numpy arrays only? I don't think we have an object dtype based on Arrow as Polars do, right?

Back to the discussion about naming, I think `pyobject[pyarrow]` and I guess `pyobject[numpy]` would be my preference. I think pyobject is more explicit and way more clear than object.
> A few API question it'd be helpful to see addressed explicitly:
> 
> 1. `ser[0] = np.nan`  Does this assign NaN or does this silently replace with pd.NA?
> 2. Same question with NaN in a list passed to the constructor.
> 3. Same questions but with None or pd.NaT
> 4. `ser = pd.Series([pd.NA, None, pd.NaT, np.nan], dtype="object_nullable")`.  (Assuming we don't silently replace) What do isna/fillna/skipna do?

what i've said in the initial draft is

> The proposed nullable object array will be
unable to hold `np.nan`, `None` or `pd.NaT` as these will be
considered missing in the constructors and other conversions
when following the existing API for the other nullable
types. Users will not be able to round-trip between the
legacy and nullable object dtypes.

So I'm assuming that to ease the transition that all these will be treated as missing, updating the mask appropriately and represented as pd.NA. So my assumption is that we do silently replace. Do you disagree with this or perhaps prefer warnings for the assignment?
> Personally I think `pd.NA` should be used for setting values in the boolean mask, everything else should go into the values and not have special treatment.
> 
> While it may be counterintuitive at first based on past behavior, I think it's the simplest to implement, to explain, and to understand.

seems reasonable. I assumed we would match the behavior of other nullable extension arrays for consistency. I'll audit this before opening the official discussion period.

Thanks for highlighting this. 
Oh one more: 

5) `ser = pd.Series([2, pd.Timedelta(1)], dtype="nullable_object")`  # what happens with `ser / 0`?
> 2\. Maybe a meta comment about this topic, I do think discussing a "nullable object type" would be better suited in a PDEP that discusses all "nullable types" (system) to avoid fragmentation/diversion of terminology, null semantics, etc. as inevitability the other types will be discussed as well.

yes I wanted to discuss this in PDEP-16 as the type mapping in the first commit showed that the traditional numpy object dtype would be retained. A few days away from a full year since that was opened and the PDEP in incomplete with no discussion. sitting on a draft PDEP is harming the project. If we can get PDEP-16 moving then I would probably not have opened this.

I think being part of the bigger discussion is crucial if the object_nullable was to become a default in the future. I did not explicitly state that in this initial draft even though I made the comment that could be interpreted as the motivation for this PDEP. As I said in https://github.com/pandas-dev/pandas/pull/61599#discussion_r2136185318 i'm happy to remove that.

The motivation is just to create a dtype consistent with the other pandas nullable dtypes. We have Int, Float, Boolean etc but do not have one for object. 
> Oh one more:
> 
> 5. `ser = pd.Series([2, pd.Timedelta(1)], dtype="nullable_object")`  # what happens with `ser / 0`?

under the current proposal these would be missing in the mask (represented as the pd.NA scalar). if we instead do what @datapythonista suggests in https://github.com/pandas-dev/pandas/pull/61599#issuecomment-2956329138 then this would yield np.nan and would be a sentinel value.


> 1. While pandas still `object` a "string dtype", would `"object_nullable"` be yet-another-string-dtype with an alternative NA semantics? I would be more comfortable with this type if pandas just considers `object` as purely PyObjects

to be compatible with pyArrow and polars?
>> While pandas still object a "string dtype", would "object_nullable" be yet-another-string-dtype with an alternative NA semantics? I would be more comfortable with this type if pandas just considers object as purely PyObjects

> to be compatible with pyArrow and polars?

Sorry had some typos in that statement. I meant to say "While pandas still considers `object` a "string dtype"".

Well there no compatibility with those two since they don't have a type to store arbitrary PyObjects. I think it's a potential "strength" that pandas can provide a "nullable object dtype" to store PyObjects with null semantics. My concern is pandas still conflating `object` with "string".

So for example, `str` APIs still work with `object` type. Would you expect `str` APIs to work with `object_nullable`?
> The motivation is just to create a dtype consistent with the other pandas nullable dtypes.

Behind a flag. Not necessarily making any commitment here to changes to the existing pandas api. To allow evaluation of the concept as we did with the other pandas nullable dtypes at first. The StringDtype has been available for a long time now before being made a pandas default type. I expect object_nullable to be experimental for a long time and to iron out any consistency issues with the other pd.NA variants of the pandas nullable dtypes and consistency issues with the legacy object dtype with the handling of values that are considered missing as @jbrockmendel highlighted.
> > > While pandas still object a "string dtype", would "object_nullable" be yet-another-string-dtype with an alternative NA semantics? I would be more comfortable with this type if pandas just considers object as purely PyObjects
> 
> > to be compatible with pyArrow and polars?
> 
> Sorry had some typos in that statement. I meant to say "While pandas still considers `object` a "string dtype"".
> 
> Well there no compatibility with those two since they don't have a type to store arbitrary PyObjects. I think it's a potential "strength" that pandas can provide a "nullable object dtype" to store PyObjects with null semantics. My concern is pandas still conflating `object` with "string".
> 

Ah i see, I interpreted that as we could have an arrow array containing the pointers to the python objects.

the nullable string array is an nullable object array with constraints. The nullable object array would not be a string array. the constraints are it's strengths as we don't have the np.nan, None, pd.NAT issue?

> So for example, `str` APIs still work with `object` type. Would you expect `str` APIs to work with `object_nullable`?

only if `object` dtype retains the str accessor. If we remove it from `object` then it would not be needed from "object_nullable"


 
> The nullable object array would not be a string array.

+1, and hopefully that can give us inspiration to decouple the base `object` dtype from meaning "string" in the future (not for this PDEP)

> the constraints are it's strengths as we don't have the np.nan, None, pd.NAT issue?

That and a general observation that users continue to store arbitrary objects (numpy arrays, pandas objects, custom classes) in pandas objects, so providing nullability to that I guess is a plus
> While pandas still object a "string dtype", would "object_nullable" be yet-another-string-dtype with an alternative NA semantics?

hopefully not.

The intention was an array with exactly the same NA semantics of the other pd.NA variants of the pandas nullable types, i.e. the original nullable string dtype, Int, Float, Bool etc. Should be **no** differences in my opinion. The issues regarding handing of np.nan are perhaps similar to the array of issue for the pandas nullable float type if we allow it. If we don't it's not an issue but the behavior of the object and object_nullable would be different. In an ideal world we would perhaps want as much backwards compatibility as possible.

If we don't have pyarrow as a required dependency, then I would expect object backed variants of the new dtypes. These would presumably be based on an nullable object array with constraints. For these cases the constraints would be their strength. I've not included that as a motivation either at this point. If the discussion here exceeds the PDEP-10/PDEP-15 discussion which I expect it to following the timescales in PDEP-1 that could be added depending on the outcome. I assuming PDEP-10 is rejected. If PyArrow is made a required dependency for 3.0 then maybe we could shift focus to a nullable object array backed by pyarrow pointers to python objects instead.
> > So for example, `str` APIs still work with `object` type. Would you expect `str` APIs to work with `object_nullable`?
> 
> only if `object` dtype retains the str accessor. If we remove it from `object` then it would not be needed from "object_nullable"

I see that discussing accessors is a glaring omission from my draft.
The main use case I have for PyObject types (there can surely be others) is for loading data that can't be casted to other types. For example, loading from a nested JSON containing heterogeneous types (thinking on Arrow structs, otherwise even with homogenous types no other type can hold it in a reasonable way).

In that sense, what it makes sense to me is the the PyObject type provides the functionality to "fix" the data, so it can be transformed and converted to another type with the specific functionality. Feels like `.map()`, `.astype()` and not much more should be enough if I'm not missing anything. I don't think `.map()` to run arbitrary Python code on Python objects is an unresonable choice, and since they are PyObjects anyway, I don't think it should be significantly slower than the accessor methods.

There can surely be other considerations, but so far +1 on not supporting `.str` (or other accessors) in the new type.
> There can surely be other considerations, but so far +1 on not supporting `.str` (or other accessors) in the new type.

Interesting idea. The proposed nullable object dtype with a list accessor would be well suited to the return type of str.split(expand=False) of the pd.NA variant of the pandas nullable string dtype.

> In that sense, what it makes sense to me is the the PyObject type provides the functionality to "fix" the data, so it can be transformed and converted to another type with the specific functionality. Feels like `.map()`, `.astype()` and not much more should be enough if I'm not missing anything. I don't think `.map()` to run arbitrary Python code on Python objects is an unresonable choice, and since they are PyObjects anyway, I don't think it should be significantly slower than the accessor methods.

That's also an interesting idea. Designing a nullable object array from scratch instead of trying to match all the functionality of the current numpy object array!
> That's also an interesting idea. Designing a nullable object array from scratch instead of trying to match all the functionality of the current numpy object array!

For context, I do think we're requiring PyArrow in pandas 3.0, and using PyArrow types by default. My comments here are based on it. Unfortunately we need to wait an extra month to find out. But spending the next two years reinventing the nullable types that PyArrow already give us for free seems a very poor investment of our time. And even if we do that, I'd do it by continuing to release 2.x versions. We'll continue this discussion in the appropriate threads, just wanted to clarify that we are probably discussing this new PyObject type from very different points of view, as PDEP-15 is keeping the direction of the project very uncertain.
> For context, I do think we're requiring PyArrow in pandas 3.0, and using PyArrow types by default.

Where has this been discussed? My understanding is that the PyArrow types are just experimental and the pandas nullable types will be the default in the future (as per PDEP-16 when its done)
This can't be discussed until there is clarity about PDEP-10 and PDEP-15 in my opinion. And it's been like 3 years of PyArrow types being experimental anyway, so I think it's something worth discussing and considering if we do require PyArrow, no?
I read the above as using PyArrow types by default in 3.0. That can't happen unless we postpone the 3.0 release and as per our deprecation policy have at least 2 minor releases in 2.x with the appropriate warnings for the breaking changes.