ENH: Implement DataFrame interchange protocol
State: closed

Do note that this PR is currently work-in-progress, mostly to facilitate the discussion on how the implementation should be going.

It also vendors the [exchange spec](https://github.com/data-apis/dataframe-api/pull/74) and [exchange tests](https://github.com/data-apis/dataframe-api/pull/75), which aren't yet merged at the consortium, so I'll keep updating the vendored copies as the discussion goes there.

More tests are also to be added, as well as the implementations of some cases (a lot of non-central cases are `NotImplemented` now, as I've built this upon [the prototype](https://github.com/data-apis/dataframe-api/blob/main/protocol/pandas_implementation.py).

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Hello @vnlitvinov! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-04-24 10:03:07 UTC
cc @jreback for preliminary feedback
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@jreback @jbrockmendel I've responded to your comments, but I suggest to refrain from re-reading the PR just yet - I'm in the middle of improving it yet further, and I'll make a comment when it's again ready for reviewing.

Thanks again for your feedback!
Okay, I think logic-wise it's ready to be reviewed.

I still need to make CI happy about code style etc., but I don't expect a lot of changes for that.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Please remove the `stale` label
This PR finally passes the code checks (and new functionality passes newly added tests which cover at least the basic usage of the API) on my end, so I'm marking this PR as "ready for review".
CI failures look like flaky tests, not related to my changes.

So I consider this PR ready for reviewing, ping @jbrockmendel @jreback 
@jreback @jbrockmendel @dchigarev @YarShev I've rebased off `main` hoping it would fix flakes in CI, please review.
@vnlitvinov, I see no answers to my questions above. Please take a look at them.
@YarShev I hope I've answered all of them now, I'm sorry I've somehow missed that you've added more responses to initial review.

@jorisvandenbossche should I rename subpackage `pandas.core.exchange` to `pandas.core.interchange` to align with new PR title?..
@YarShev 

> Should this check be put in df.dataframe() to get an error before playing around with the dataframe implementing the protocol?

_I cannot find this comment somewhere where I can write an answer, so I'm going to type it as a general comment._

I think this check should be delayed as much as possible because it's potentially scanning all the items in the column, which is a heavy operation while a user might just be needing some small amount of information (or might be wanting to get some particular column but not this string/object one).
> @YarShev
> 
> > Should this check be put in df.dataframe() to get an error before playing around with the dataframe implementing the protocol?
> 
> _I cannot find this comment somewhere where I can write an answer, so I'm going to type it as a general comment._
> 
> I think this check should be delayed as much as possible because it's potentially scanning all the items in the column, which is a heavy operation while a user might just be needing some small amount of information (or might be wanting to get some particular column but not this string/object one).

This is about handling string and object dtype. Let's continue the discussion there ([link](https://github.com/pandas-dev/pandas/pull/46141#discussion_r856449857).)
should there be tests that the protocol is round-trippable? e.g.

```
tm.assert_frame_equal(df, pd.api.exchange.from_dataframe(df.__dataframe__()))
```

for some/most of possible dfs? (e.g. empty, various types), if they have a non-range index they should raise? what about non-string columns names?

can certainly do this in another PR as well.
There already are a few:

https://github.com/pandas-dev/pandas/blob/cc94e57a886e1bc638e913d1196bf49cd324f73c/pandas/tests/exchange/test_impl.py#L53-L68

and

https://github.com/pandas-dev/pandas/blob/cc94e57a886e1bc638e913d1196bf49cd324f73c/pandas/tests/exchange/test_impl.py#L71-L90

Maybe I should extend the second one and take a subset of pandas DataFrame using same indices and compare it with the one obtained via protocol...
> if they have a non-range index they should raise? what about non-string columns names?
> 
> can certainly do this in another PR as well.

I would rather make it in a separate PR, as this one is already big...
> > if they have a non-range index they should raise? what about non-string columns names?
> > can certainly do this in another PR as well.
> 
> I would rather make it in a separate PR, as this one is already big...

no for sure, pls create a todo issue (and PRs)!

thanks for all of this @vnlitvinov and @YarShev for all the review!