ENH: Allow third-party packages to register IO engines
State: closed

- [X] xref #61584 
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Added the new system to the Iceberg connection only to keep this smaller. The idea is to add the decorator to all other connectors, happy to do it here or in a follow up PR.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: API Design

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61642/
@mroeschke I addressed your comments and I think this should be ready when you've got time. Thanks!
Fully agree, I just didn't want to make this PR too big by allowing the engines and adding`**kwargs` everywhere here. I'll do it in a follow up.

@pandas-dev/pandas-core any opinion or comment before merging this?
Thanks for the ping. I haven't been involved enough to really block, but I'm curious what the advantage of this is over leveraging the Arrow PyCapsule interface for data exchange; I feel like the latter would be a better thing to build against, given it is a rather well adopted standard in the ecosystem
Good point, thanks for the feedback. I think this is a higher interface that still allows for using the Arrow pycapsule internally. Surely an option would be to get rid of I/O in pandas, and have an ecosystem of readers that can be used via a single `pd.read(engine)` or something similar. But I think it's better to use the current API, allow third party engines as implemented here, and let the engines decide how the data is exchanged. Or what is your idea?
It would depend on a resolution to https://github.com/pandas-dev/pandas/issues/59631, but for demo purposes let's assume we decide to implement a new `DataFrame.from_pycapsule` class method.

So instead of an I/O method like:

```python
df = pd.read_iceberg(..., kwargs)
```

You could construct a dataframe like:

```python
df = pd.DataFrame.from_pycapsule(
    # whatever the iceberg API is here to create a table
    pyiceberg.table(..., kwargs)
)
```

The main downside is verbosity, but the upsides are:

  1. Generic approach to read any datasource
  2. pandas itself has to do very little development (third parties are responsible for integration)
  3. polars, cudf, etc... all benefit in kind

So yea this could be extended to say even Delta if they decided to implement the PyCapsule interface (whether they do currently or not, I don't know):

```python
df = pd.DataFrame.from_pycapsule(
    # whatever the delta API is here to create a table
    DeltaLake.table(..., kwargs)
)
```

and if polars decided on the same API you could create that dataframe as well:

```python
df = pl.DataFrame.from_pycapsule(
    # whatever the delta API is here to create a table
    DeltaLake.table(..., kwargs)
)
```
As I said though, I haven't been involved enough to really block, so if this PR has some support happy to roll with it and can clean up later if it comes to it. Thanks for giving it consideration @datapythonista 
i don't know anything about pycapsules.  is that effectively a pd.from_arrow_table method that isn't pyarrow-specific?
Yea kind of. In a generic sense, a PyCapsule is just a Python construct that can wrap generic memory. The Arrow PyCapsule interface further defines lifetime semantics for passing Arrow data across a PyCapsule:

https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html

Somewhat separately there is the question of how you would read and write the data in a capsule. For pandas that is pyarrow, but other libraries may choose a tool like nanoarrow for a smaller dependency
@WillAyd what you propose seems reasonable, but I guess we aren't planning to remove all pandas IO anytime soon. And if we keep our readers and writers with multiengine support, I think this interface is going to be useful, even if long term we move into the pyarrow capsule reader you propose. Also, since pandas is not arrow based yet, this PR could be used to move the xarray connectors to the xarray package, while using pycapsule wouldn't be ideal for pandas/xarray interchange, as they are numpy based.
> And if we keep our readers and writers with multiengine support

ATM that's just csv and parquet?  And the parquet one plausibly is not needed?

AFAICT this adds a bunch of new tests/code/docs, complicates the import with entrypoints, and lets 3rd parties hijack our namespace.  All when there's a perfectly good option of using their own namespace.

Also if we ever did change defaults like #61618 or PDEP16, that would Break The World for any 3rd party readers that we are implicitly committed to supporting.

-1.
> Also, since pandas is not arrow based yet, this PR could be used to move the xarray connectors to the xarray package, while using pycapsule wouldn't be ideal for pandas/xarray interchange, as they are numpy based.

Definitely not an expert, but I want to point out that DLPack also offers a PyCapsule for data exchange. See https://dmlc.github.io/dlpack/latest/python_spec.html

So depending on how generic we want things to be, PyCapsule support doesn't _just_ mean consuming Arrow data, but _could_ mean dlpack data as well (which I assume xarray can do, if it doesn't already)
> ATM that's just csv and parquet? And the parquet one plausibly is not needed?

Any reader could have an engine option, this would allow having xarray, sas... as other packages.

> AFAICT this adds a bunch of new tests/code/docs, complicates the import with entrypoints, and lets 3rd parties hijack our namespace. All when there's a perfectly good option of using their own namespace.

This adds minimal tests, code or docs, any other PR adds as many as this. We already allow entrypoints in pandas, and 3rd parties can not use this to change the namespace directly, just to allow `engine="foo"`.

> Also if we ever did change defaults like https://github.com/pandas-dev/pandas/issues/61618 or PDEP16, that would Break The World for any 3rd party readers that we are implicitly committed to supporting.

We could add the value of the flag setting the types to use to the interface, so third parties can transition in the same way as us.

> -1.

Being honest I think you'll block anything that is Bodo related. I think it was best to be -1 to accept their money when they offered it. In any case, I'll close this, and I'll open a separate issue to discuss returning the remaining funds, as I think it can make sense at this point.
I'm not blocking, just voting against.  If everyone else disagrees, I'll make my peace with it.

I'm not against everything bodo-related, but I do default pretty hard against letting 3rd parties piggyback on our namespace.
it might be nice to sequester all third-party integrations into an `external` (or `thirdparty`) namespace, like `pd.external.read_iceberg` (and something like `pd.DataFrame.external.to_iceberg`).
> I'm not blocking, just voting against. If everyone else disagrees, I'll make my peace with it.

I apologize @jbrockmendel as I think I overreacted. While I don't understand or share your concerns, I think it's good to have your feedback.

The problem is that I think the new pandas governance, instead of fixing any of our decision making problems, made them even worst. While your -1 vote is fair and just a point of view that I appreciate having, in practice means that this PR is dead unless I try the PDEP path and wait the 3 months time window, which I won't do.

I'll leave this open for a bit longer, just in case there is enough support on this to make you be ok with it. And otherwise I'll close later when it becomes stale.
> While I don't understand or share your concerns

Not sharing them is fine, but if you don't understand that means I've done a poor job explaining them, so I'll try again:

Zero users have asked for this.  There are zero use cases in which `bodo.read_iceberg` is not a fully-functional (and clearer!) alternative.  To my mind, that means there are zero upsides.

On the other hand, this complicates are API, which users frequently _do_ complain about.  Sure it doesn't complicate it _much_, but these things add up.  Similarly more docs and more code and more tests increase burdens, and these things add up.  These are very real downsides.

But the part that I actually think will come to bite us is that it muddles responsibility.  Users won't know what we do and don't maintain, and will complain to us.  If we want to change something that breaks a 3rd party reader, people will argue that needs a deprecation process.  There will be crappy poorly-implemented engines out there with _our_ name associated with them.

All downsides, no upsides.

> instead of fixing any of our decision making problems, made them even worst

No argument there.  FWIW I'm pretty comfortable being the "that doesn't need a full PDEP" guy.
> FWIW I'm pretty comfortable being the "that doesn't need a full PDEP" guy.

As the person usually asking for a PDEP, I don't think one is needed here.

But @jbrockmendel brings up some valid concerns, and I leave it to others to determine whether those concerns mean we don't accept this addition to pandas.

Thanks @jbrockmendel, this is helpful, I understand better your concerns, and I actually agree in two of your point.

While I think this is actually a very small change, it wouldn't be worth if the goal was to allow Bodo (or others) to write a reader. We got rid of `read_gbq` long ago, because maintenance of just the wrapper was annoying, and also not an open source format, but a paid infra. They implemented `pandas_gbq.read_gbq`, and no big deal. That was a whole format, not an engine, but we could do the same with anything.

I just find this has some advantages:

- For writers, what you propose doesn't support method chaining
- With your proposal, if I want to get rid of the xarray IO  code in pandas, it's a breaking change. With this proposal I can move the wrapper to xarray, add the entrypoint, and users with xarray installed won't see a difference. Users without xarray may see a different error message if using xarray. And our CI wouldn't need to test for xarray, and more important, our environment wouldn't have xarray. If we can do this with few engines, I'll probably won't have to continue spending countless hours fighting with the problems of the conda solver.
- If our CSV readers (or Excel readers) were moved out to other packages, with your proposal their signatures would become independent. And changing from one engine to another would require changing the module, possibly the function name, and the parameters. With this PR it'd be just the engine name, and parameters only if engine specific parameters exist. For context, I'd like to have pandas CSV engines for Polars and DuckDB, and I'd like that the code lives in those projects. This PR (and the needed follow up to support all formats) would allow it from the pandas side.

Second point I agree is that users may not immediately underatand that the engines don't live in pandas. I don't have a solution, but two things to consider.

- For most connectors the code already lives in an external package, we just maintain a thin wrapper. So, in a way, we already take responsibility for other packages code.
- The docs won't mention external engines, but the API docs can mention external engines are supoorted, as well as the user guide. For new engines, I think users will have to learn them from the other packages docs, or our ecosystem page. So, except for the users that read code written by someone else, I think most users can become familiar with the idea that enginea are not part of pandas.
> For writers, what you propose doesn't support method chaining

A user who is heroin-level obsessed with method chaining can use `.pipe(bodo.to_iceberg)`

> if I want to get rid of the xarray IO code in pandas

If that is the real goal here, please just make an issue for that.  In that scenario, I would _also_ say that the relevant reader/writer belongs in its own namespace.
@twoertwein we discussed supporting what you mention in PDEP-9. And if people aren't convinced to have this for engines only, I don't think there can be consensus for supporring aebitrary formats in the pandas namespace.
> If that is the real goal here, please just make an issue for that.

I wouldn't say it's the real goal, but surely one of the main reasons. I wrote PDEP-9 to go into the details on why I think pandas IO should work as Python modules. 

I think the main reason why people use Python is because code is readable, and it's batteries included via pip/conda. I'd say people use pandas because it's a Swiss knife, also with everything included. If a user in Python is asked to implement code in a C extension, is like telling them it's not possible, because they are used to pip install + import. In a similar way (in my opinion), telling users to import another module to read, and pipe to write, is telling them the reader is not supported by pandas. Surely the difficulty is not comparable to writing a C extension, but the feeling that is not supported and that a hack is needed are probably the same.

The real goal here is to reduce the gap between a pandas core IO connector, and an external IO connector. To the same as a standard library package and a cheeseshop package. And one of the main motivations is that moving IO connectors into and out of pandas would become trivial, both technically and in terms of backward compatibility. PDEP-9 tried that fully, this is just for engines of pandas supported formats. But same idea, just that this PR is trivial, both in code and conceptually, and PDEP-9 came with problems of naming conflicts, pollution of the pandas namespace.

But I personally don't think discussing again PDEP-9 is needed. I think it's mostly whether the advantages here are worth the added complexity. To me that's an absolute yes. I guess you don't see the advantages as significant as you think it's fine to just use pandas modules and pipe. I disagree with that, but it's surely a valid point of view. In practice we won't move any IO connector out of pandas with this PR. But it's surely not clear if that was going to happen anyway, so not an immediate advantage.
@Dr-Irv you are still blocking this PR. Is it that you want it to be blocked, or that you forgot to remove the requested change flag? From your last comment I can't tell which of Brock's comments you share, amd if they are a blocker. But if you just didn't forgot to remove the flag, I don't think it's very nice to block someone's work without being clear what change is expected, or why this shouldn't be merged in any form.
> is telling them the reader is not supported by pandas

Correct.  It's implemented and maintained by a 3rd party.  That is the correct message to send.

> To the same as a standard library package and a cheeseshop package

Using a 3rd party engine via their own namespace is _literally_ using a cheeseshop package.  

> or why this shouldn't be merged in any form

Regardless of whether Irv removes the block, please do not self-merge.
> @Dr-Irv you are still blocking this PR. Is it that you want it to be blocked, or that you forgot to remove the requested change flag? From your last comment I can't tell which of Brock's comments you share, amd if they are a blocker. But if you just didn't forgot to remove the flag, I don't think it's very nice to block someone's work without being clear what change is expected, or why this shouldn't be merged in any form.

I have comments above that still should be addressed:

- https://github.com/pandas-dev/pandas/pull/61642#discussion_r2173484073 which is suggesting that `pyproject.toml` is required to use this feature.  
- https://github.com/pandas-dev/pandas/pull/61642#discussion_r2173486073 which is that I think the `whatsnew` is unneeded (or should be changed)
- https://github.com/pandas-dev/pandas/pull/61642#discussion_r2173157243 which is about moving the `load` operation to when the entry point is actually needed.

> Thanks @Dr-Irv for your comment, and thanks for reminding me that you'll continue to block other people's work based on zero impact and extremely opinionated details that I'd bet only make sense to you.

I'm sorry that you see my comments in that light.  That's certainly not my intent.  

> 
> I guess my options are letting you blackmail me so your ego is happy, report your behaviours to the CoC or the steering committe you are part ot it. Or stop wasting my time in the toxic environment pandas has become. I think the choice is clear, so take this as a good bye.

I am not trying to blackmail you.  I am providing what I believe to be constructive comments to make the work you do understandable by others.  


I wanted to add my perspective on this PR. I work at Bodo, but before that I was a user of Pandas myself.

I think that having the engine argument in Pandas as well as the corresponding docs page offers an easy way for users to solve their problems without researching other packages or workarounds. For example, reading multiple csv files from a directory. A Pandas user might write code that looks like this:
``` py
import glob
import pandas as pd

path = "my_dir"
filenames = glob.glob(path + "/*.csv")
dfs = []

for file in glob.glob(path + "/*.csv"):
    dfs.append(pd.read_csv(file))

pd.concat(dfs, ignore_index=True)
```
But with the engine param and corresponding docs, they could discover changing the engine might simplify this workflow:
``` py
import pandas as pd

pd.read_csv("my_dir/*.csv", engine="some_engine")
```
Of course if they truly were bothered by this they could do some googling or ask AI and find another package that fits their use case a bit better, but I feel like Pandas should lead users in the right direction. 
> I have comments above that still should be addressed:

@datapythonista Just to clarify, I'm not for or against adding this particular feature.  The comments that I was hoping you would respond to were about documentation and performance.  With respect to the documentation, I think it can be improved (and would be helpful if this were to be reopened and merged in).  With respect to performance, you acknowledged that delayed loading is a good idea, but you didn't want to do it because you didn't think the PR would be accepted.

I'm not making the decision of accepting and merging the PR if you were to reopen it.  I will leave that to others.
> But with the engine param and corresponding docs, they could discover changing the engine might simplify this workflow:

1) I don't think we're putting the docs for third party engines in our docs.
2) A third party engine with behavior significantly different from the pandas API doubly doesn't belong in the pandas namespace.