Add DatasetDict.to_pandas
State: closed | Author: lhoestq (MEMBER)

From discussions in https://github.com/huggingface/datasets/issues/5189, for tabular data it doesn't really make sense to have to do
```python
df = load_dataset(...)["train"].to_pandas()
```
because many datasets are not split.

In this PR I added `to_pandas` to `DatasetDict` which returns the DataFrame:

If there's only one split, you don't need to specify the split name:
```python
df = load_dataset(...).to_pandas()
```
EDIT: and if a dataset has multiple splits:
```python
df = load_dataset(...).to_pandas(splits=["train", "test"])
# or
df = load_dataset(...).to_pandas(splits="all")

# raises an error because you need to select the split(s) to convert
load_dataset(...).to_pandas()
```

I do have one question though @merveenoyan @adrinjalali @mariosasko:

Should we raise an error if there are multiple splits and ask the user to choose one explicitly ?



Comments: The current implementation is what I had in mind, i.e. concatenate all splits by default.

However, I think most tabular datasets would come as a single split. So for that usecase, it wouldn't change UX if we raise when there are more than one splits.

And for multiple splits, the user either passes a list, or they can pass `splits="all"` to have all splits concatenated.

I think it's better to raise an error in cases when there are multiple splits but no split is specified so that users know for sure with which data they are working. I imagine a case when a user loads a dataset that they don't know much about (like what splits it has), and if they get a concatenation of everything, it might lead to incorrect processing or interpretations and it would be hard to notice it.
("explicit is better than implicit")

I just changed to raise an error if there are multiple splits. The error shows an example of how to choose a split to convert.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5312). All of your documentation changes will be reflected on that endpoint.

Thanks for the review, I've updated the type hint and added a line to raise an error on bad splits :)

Merging https://github.com/huggingface/datasets/pull/5301 would eliminate the need for this PR, no?

In the meantime, I find the current API cleaner.

This solution is simpler than https://github.com/huggingface/datasets/pull/5301 and covers most cases for tabular datasets, so I'm in favor of merging this one and put https://github.com/huggingface/datasets/pull/5301 on stand by

Let me know if it sounds good to you @mariosasko @albertvillanova :)

I'm still not convinced. If `DatasetDict` needs this method and there is no other way, then IMO it would make more sense to return a dictionary with the splits converted to `pd.DataFrame`. 

@mariosasko the issue we're dealing with is that in tabular scenarios, we often don't have splits in the dataset, and imposing that concept to people dealing with the library hampers adoption.

@adrinjalali This PR proposes a solution inconsistent with the existing API (in other words, a solution that clutters our API 🙂). Moreover, our library primarily focuses on larger-than-RAM datasets, and tabular datasets don't (directly) fall into this group.

Instead of the temporary "fix" proposed here, it makes much more sense to align `load_dataset` with both tabular and DL workflows "in a consistent way", so I suggest we continue our discussion from https://github.com/huggingface/datasets/issues/5189 to have this resolved by version 3.0.

closing this one for now