ENH: Add the support of the Delta Lake format in Pandas as an optional extra dependency
State: closed

- [x] closes #40573
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

[Delta Lake](https://delta.io/) is an open-source storage framework, a Python library [delta-rs](https://github.com/delta-io/delta-rs) is available to access it using Pandas. Integrating `deltalake` in pandas as an optional extra dependency to pandas makes the support work out of the box for pandas users.


Labels:

Comments:
Hello @jreback 👋,

Thank you for your review!

I included Delta Lake as a new API file because (by definition) it uses a transaction log with its [PROTOCOL](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) to read different files format, including data in parquet. I don't have a strong opinion on this; I was asking the question myself actually: what would be the best place for this format's integration?

If a new [engine](https://github.com/pandas-dev/pandas/blob/main/pandas/io/parquet.py#L47) in parquet is the best place for you, I could easily move the integration in this part, WDYT?  
I would also be -1 to adding a dedicated API for this as well.

Additionally, since this functionality is essentially `deltalake.DeltaTable(**kwargs).to_pandas(**kwargs)`, the simplicity of this functionality makes me skeptical this needs to live in pandas as it would incur testing and dependency management overhead. 

Maybe it may be more appropriate to add this to the ecosystem docs?
Hello @mroeschke 👋,

Thanks for the input, this [PR](https://github.com/pandas-dev/pandas/pull/40636) already included `deltalake` to the ecosystem docs. I was inspired by the other apis implementation.

Should I close this PR or move the integration into another `engine` option in parquet? 
I think the advantage of adding this is that Delta Lake is arguably the best way to read data into a pandas DataFrame.  Disclosure: I am on the Delta Lake team.

The less data you load into a pandas DataFrame, the better.  Delta Lake lets users easily load less data into pandas DataFrames (via column pruning, file skipping from metadata, and predicate pushdown filtering).  pandas only allows for column pruning via `read_parquet(columns="")`, but there is a lot more data skipping that could be provided to end users.

pandas `to_parquet` has a `partition_cols` argument that lets users take advantage of disk partitioning on the write side, but there isn't any `from_parquet` argument that'd let them take advantage of file skipping on the read side.  See the [Dask read_parquet filters argument](https://docs.dask.org/en/stable/generated/dask.dataframe.read_parquet.html) for more the type of interface that could be exposed.  Delta Lake could fill the gap and let users take advantage of data skipping for partitioned datasets.

I'd argue that the Z ORDERING offered by Delta Lake is better than the disk partitioning currently offered by pandas.

Another "why isn't this just parquet" argument is that Delta Lake allows for schema enforcement that Parquet does not provide.  All the pandas data source connectors are schema-on-read.  It'd be nice for pandas users to have a schema-on-write connector.

I wrote blog posts on [reading Delta Lakes into pandas DataFrames](https://mungingdata.com/pandas/read-delta-lake-dataframe/) and [how to version you data / time travel with pandas](https://delta.io/blog/2022-10-15-version-pandas-dataset/) with some more background info.  

If this gets added, we should probably add the `filters` argument.  I do think this would provide pandas users with a fundamentally better way to read data and would give them the data skipping capabilities they can get with other query engines.  I think these features are especially pertinent for pandas users because of the strict pandas memory limits.  I can also see the argument that this should be kept separate and would like to reiterate my bias here.  Thank you.
> Thanks for the input, this https://github.com/pandas-dev/pandas/pull/40636 already included deltalake to the ecosystem docs. I was inspired by the other apis implementation.

Ah okay I didn't realize this was already in the ecosystem docs.
Agree with existing -1 comments here - we already have a pretty exhaustive API that we are very conservative with when it comes to additions. I think @mroeschke suggestion covers the functionality
Thanks for the pull request, but since we have a few -1's here from the core dev team it appears there not much appetite to maintain a dedicated function for this format, so closing.