Implement first-class List type
State: closed

Quick POC for now. There's a lot to do here but hoping to work in pieces. This currently assumes pyarrow is installed.

The blocks / formatting stuff is not super familiar to me so hoping @mroeschke or @jbrockmendel might have some ideas on how to better approach. I think the main problem I am having is the Block seems to want to infer the type from the values contained. That works for NumPy, but doesn't work with PyArrow, for example when you have an array of all `nulls` that is separately paired with a type of`list[string]` 

Labels:
name: Stale

Comments:
Does this assume moving forward with the logical type system PDEP? i.e. List type backed by multiple (in theory) implementations
PDEP-14 would be nice but I don't think its required here. If we do not revert PDEP-10, then we can assume pyarrow is required and just build off of that. This can fit logically into the extension type system. 

We may just want to start referring to that as something else besides "numpy_nulllable," but there is an issue already open for that https://github.com/pandas-dev/pandas/issues/59032
> The blocks / formatting stuff is not super familiar to me so hoping @mroeschke or @jbrockmendel might have some ideas on how to better approach. I think the main problem I am having is the Block seems to want to infer the type from the values contained.

Yah I'd really rather avoid the changes this makes in that part of the code.  Will comment in-line and see if we can find alternatives.
the thing ive always assumed would be a PITA with a ListDtype is `__setitem__` distinguishing whether `df.iloc[x, :3] = [a, b, c]` is to behave like

```
df.iloc[x, 0] = a
df.iloc[x, 1] = b
df.iloc[x, 2] = c
```

vs

```
df.iloc[x, 0] = [a, b, c]
df.iloc[x, 1] = [a, b, c]
df.iloc[x, 2] = [a, b, c]
```

have you given any thought to that?
For a List data type the first option wouldn't be possible, since those are scalars values. So I think the latter is correct; if you wanted unpacking I think you'd need to provide a list of lists
Is the idea for this to replace `ArrowDtype(pa.list_etc())` and the corresponding ArrowEA or for both to exist?  Having multiple implementations of things has been a pain point for the StringDtype(s).
> Is the idea for this to replace `ArrowDtype(pa.list_etc())` and the corresponding ArrowEA or for both to exist? Having multiple implementations of things has been a pain point for the StringDtype(s).

Yea I'd prefer this to be the user-facing API for working with list data, but having that fully replace the `ArrowDtype(pa.list_())` is a PDEP-13 discussion

There's a few places in I/O where this can work its way into the `numpy_nullable` type system, i.e. when trying to serialize lists to/from SQL databases and parquet
@WillAyd  `Series.str.split()` and `Series.str.rsplit()` can return `List` values, so that should be used here.
@Dr-Irv that's a great idea although I'm hesitant to change the API of any existing functions without taking PDEP-13 further
> @Dr-Irv that's a great idea although I'm hesitant to change the API of any existing functions without taking PDEP-13 further

I don't think it's an API change.  It's just a question of the `dtype` of the result of `str.split()` and `str.rsplit()` when `expand=False`.  With your PR, you could make that result have `ListDtype` if `pyarrow` is installed, otherwise return object dtype (which is what happens today)

That's an interesting idea. My main concern would be the portability of pandas, i.e. running code in one environment without pyarrow installed will yield a potentially different code path than what what you would get in an environment with pyarrow. I think the logical type abstraction would go a long ways to simplify that for end users.

On the flip side, we are still somewhat in limbo as to whether or not pyarrow is actually required for pandas 3.0, given the status of PDEP-15 (we've also discussed this on Slack). If we hard commit back to pyarrow, then you could implement this with less friction I think

Appreciate the input - I think there's a lot of potential here so just need to decide as a team where we feel comfortable with non-NumPy originated types in the default space
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Not actively working on this - let's clear the queue