Allow downloading just some columns of a dataset
Opened by osanseviero (open)

**Is your feature request related to a problem? Please describe.**
Some people are interested in doing label analysis of a CV dataset without downloading all the images. Downloading the whole dataset does not always makes sense for this kind of use case

**Describe the solution you'd like**
Be able to just download some columns of a dataset, such as doing
```python
load_dataset("huggan/wikiart",columns=["artist", "genre"])
```

Although this might make things a bit complicated in terms of local caching of datasets.

Comments:
In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess

Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.

Bumping the visibility of this :) Is there a recommended way of doing this?

Passing `columns=[...]` to `load_dataset()` in streaming mode does work if the dataset is in Parquet format, but for other formats it's either not possible or not implemented

I tried using the `columns=['bambara']` on this dataset `oza75/bambara-tts` which is in parquet, but it does not work. This feature is really useful because sometimes you don't want to download the whole dataset but just a few columns.

It doesn't work for the dataset with `parquet` format. Are we missing something?

It only works for `streaming=True`. When not streaming it does download the full files locally before reading the data

Hi @lhoestq, I have an audio dataset of 250GB on the huggingface hub in parquet format. I only wanted to load the text column. It is taking a lot of time. It seems like it is downloading audio as well even in streaming mode. 

bump on this

Something like this worked for me:
```py
ds = load_dataset(
    "parler-tts/libritts_r_filtered",
    "clean",
    streaming=True,
    columns=['text_normalized']
)
```

I want to get all the captions from this dataset https://huggingface.co/datasets/gmongaras/Imagenet21K_Recaption
and annoyingly you need 2x the disk space listed (3.8tb downloaded, and then it needs another 3.8 tb when it does the "generating train split") to download and load the dataset using load_dataset. So that made it unfeasible for me, I even wasted a few hours trying to do it on a rented instance with 5tb of disk space (which i thought would be more than enough, but isnt because of the 2x disk space thing)

Doing it in streaming mode is way too slow, going through all the rows will take ages. Please make a simpler solution to just get a few columns

You stream the dataset and speed it up with multiple workers, can you try this ?

```python
from datasets import load_dataset
from torch.utils.data import DataLoader

ds = load_dataset("gmongaras/Imagenet21K_Recaption", split="train", streaming=True, columns=["recaption"])
dl = DataLoader(ds, num_workers=8)

for example in dl:
    ...
```

You can also be interested in checkpointing/resuming the streaming (e.g. in case of connection error), check this [guide](https://huggingface.co/docs/datasets/v3.3.0/en/use_with_pytorch#checkpoint-and-resume) for more details

Alternatively feel free to try with other tools that can load Parquet and select columns, like DuckDB / Polars / Spark (see the list of integrated libraries [here](https://huggingface.co/docs/hub/datasets-libraries)).

> You stream the dataset and speed it up with multiple workers, can you try this ?
> 
> from datasets import load_dataset
> from torch.utils.data import DataLoader
> 
> ds = load_dataset("gmongaras/Imagenet21K_Recaption", split="train", streaming=True, columns=["recaption"])
> dl = DataLoader(ds, num_workers=8)
> 
> for example in dl:
>     ...
> You can also be interested in checkpointing/resuming the streaming (e.g. in case of connection error), check this [guide](https://huggingface.co/docs/datasets/v3.3.0/en/use_with_pytorch#checkpoint-and-resume) for more details
> 
> Alternatively feel free to try with other tools that can load Parquet and select columns, like DuckDB / Polars / Spark (see the list of integrated libraries [here](https://huggingface.co/docs/hub/datasets-libraries)).

the dataloader is faster but not by much. I get 15 its/s. Will take like 270 hours at this pace.

Can you check your connection ? I'm at 1,000+ it/s on colab and also on my (consumer) internet connection.

You can also try to update `datasets`, `pyarrow`, `fsspec` and `huggingface_hub`