SurAyush/github_issues huggingface dataset:
Issue: Support proxy and user-agent in fsspec calls
State: open
Author: lhoestq

Since we switched to the new HfFileSystem we no longer apply user's proxy and user-agent.

Using the HTTP_PROXY and HTTPS_PROXY environment variables works though since we use aiohttp to call the HF Hub.

This can be implemented in `_prepare_single_hop_path_and_storage_options`.

Though ideally the `HfFileSystem` could support passing at least the proxies

Comments:
hii @lhoestq can you assign this issue to me?


You can reply "#self-assign" to this issue to automatically get assigned to it :)
Let me know if you have any questions or if I can help

#2289 

Actually i am quite new to figure it out how everything goes and done 

> You can reply "#self-assign" to this issue to automatically get assigned to it :)
> Let me know if you have any questions or if I can help

when i wrote #self-assign it automatically got converted to some number is it correct or i have done it some wrong way, I am quite new to open source thus wanna try to learn and explore it

#2289 #self-assign 

Ah yea github tries to replace the #self-assign with an issue link. I guess you can try to copy-paste instead to see if it works

Anyway let me assign you manually

thanks a lot @lhoestq ! though i have a very lil idea of the issue, i am new. as i said before,  but gonna try my best shot to do it.
can you please suggest some tips or anything from your side, how basically we approach it will be really helpfull.
Will try my best!

The HfFileSystem from the `huggingface_hub` package can already read the HTTP_PROXY and HTTPS_PROXY environment variables. So the remaining thing missing is the `user_agent` that the user may include in a `DownloadConfig` object. The user agent can be used for regular http calls but also calls to the HfFileSystem.

- for http, the `user_agent` isn't passed from `DownloadConfig` to `get_datasets_user_agent` in `_prepare_single_hop_path_and_storage_options` in `streaming_download_manager.py` so we need to include it
- for HfFileSystem I think it requires a PR in https://github.com/huggingface/huggingface_hub to include it in the `HfFileSystem.__init__`
Issue: Loading column subset from parquet file produces error since version 2.13
State: closed
Author: kklemon

### Describe the bug

`load_dataset` allows loading a subset of columns from a parquet file with the `columns` argument. Since version 2.13, this produces the following error:

```
Traceback (most recent call last):                      
  File "/usr/lib/python3.10/site-packages/datasets/builder.py", line 1879, in _prepare_split_single
    for _, table in generator:
  File "/usr/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 68, in _generate_tables
    raise ValueError(
ValueError: Tried to load parquet data with columns '['sepal_length']' with mismatching features '{'sepal_length': Value(dtype='float64', id=None), 'sepal_width': Value(dtype='float64', id=None), 'petal_length': Value(dtype='float64', id=None), 'petal_width': Value(dtype='float64', id=None), 'species': Value(dtype='string', id=None)}'
```

This seems to occur because `datasets` is checking whether the columns in the schema exactly match the provided list of columns, instead of whether they are a subset.

### Steps to reproduce the bug

```python
# Prepare some sample data

import pandas as pd

iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')
iris.to_parquet('iris.parquet')

# ['sepal_length', 'sepal_width', 'petal_length', 'petal_width',  'species']
print(iris.columns)

# Load data with datasets

from datasets import load_dataset

# Load full parquet file
dataset = load_dataset('parquet', data_files='iris.parquet')

# Load column subset; throws error for datasets>=2.13
dataset = load_dataset('parquet', data_files='iris.parquet', columns=['sepal_length'])
```

### Expected behavior

No error should be thrown and the given column subset should be loaded.

### Environment info

- `datasets` version: 2.13.0
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Comments:
Issue: Documentation links to examples are broken
State: closed
Author: david-waterworth

### Describe the bug

The links at the bottom of [add_dataset](https://huggingface.co/docs/datasets/v1.2.1/add_dataset.html) to examples of specific datasets are all broken, for example

- text classification: [ag_news](https://github.com/huggingface/datasets/blob/master/datasets/ag_news/ag_news.py) (original data are in csv files)

### Steps to reproduce the bug

Click on links to examples from latest documentation

### Expected behavior

Links should be up to date - it might be more stable to link to https://huggingface.co/datasets/ag_news/blob/main/ag_news.py

### Environment info

dataset v1.2.1

Comments:
These docs are outdated (version 1.2.1 is over two years old). Please refer to [this](https://huggingface.co/docs/datasets/dataset_script) version instead.

Initially, we hosted datasets in this repo, but now you can find them [on the HF Hub](https://huggingface.co/datasets) (e.g. the [`ag_news`](https://huggingface.co/datasets/ag_news/blob/main/ag_news.py) script)

Sorry I thought I'd selected the latest version.
Issue: load_dataset hangs on WSL
State: closed
Author: Andy-Zhou2

### Describe the bug

load_dataset simply hangs. It happens once every ~5 times, and interestingly hangs for a multiple of 5 minutes (hangs for 5/10/15 minutes). Using the profiler in PyCharm shows that it spends the time at <method 'connect' of '_socket.socket' objects>. However, a local cache is available so I am not sure why socket is needed. ([profiler result](https://ibb.co/0Btbbp8))

It only happens on WSL for me. It works for native Windows and my MacBook. (cache quickly recognized and loaded within a second).



### Steps to reproduce the bug

I am using Ubuntu 22.04.2 LTS (GNU/Linux 5.15.90.1-microsoft-standard-WSL2 x86_64)

Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
>>> import datasets
>>> datasets.load_dataset('ai2_arc', 'ARC-Challenge')  # hangs for 5/10/15 minutes

### Expected behavior

cache quickly recognized and loaded within a second

### Environment info

Please let me know if I should provide more environment information.

Comments:
Even if a dataset is cached, we still make requests to check whether the cache is up-to-date. [This](https://huggingface.co/docs/datasets/v2.13.1/en/loading#offline) section in the docs explains how to avoid them and directly load the cached version.

Thanks - that works! However it doesn't resolve the original issue (but I am not sure if it is a WSL problem)

We use `requests` to make HTTP requests (and `aiohttp` in the streaming mode), so I don't think we can provide much help regarding the socket issue (it probably has something to do with WSL).
Issue: `map` function doesn't fully utilize `input_columns`.
State: closed
Author: kwonmha

### Describe the bug

I wanted to select only some columns of data.
And I thought that's why the argument `input_columns` exists.
What I expected is like this:
If there are ["a", "b", "c", "d"] columns, and if I set `input_columns=["a", "d"]`, the data will have only ["a", "d"] columns.

But it doesn't select columns.
It preserves existing columns.
The main cause is `update` function of `dictionary` type `transformed_batch`.

https://github.com/huggingface/datasets/blob/682d21e94ab1e64c11b583de39dc4c93f0101c5a/src/datasets/iterable_dataset.py#L687-L691

`transformed_batch` gets all the columns by `transformed_batch = dict(batch)`.
Even `function_args` selects `input_columns`, `update` preserves columns other than `input_columns`.
I think it should take a new dictionary with columns in `input_columns` like this:
```
# transformed_batch = dict(batch)
# transformed_batch.update(self.function(*function_args, **self.fn_kwargs)

# This is what I think correct.
transformed_batch = self.function(*function_args, **self.fn_kwargs)
```

Let me know how to use `input_columns`.

### Steps to reproduce the bug

Described all above.

### Expected behavior

Described all above.

### Environment info

datasets: 2.12
python: 3.8

Comments:
Issue: Argument type for map function changes when using `input_columns` for `IterableDataset`
State: closed
Author: kwonmha

### Describe the bug

I wrote `tokenize(examples)` function as an argument for `map` function for `IterableDataset`.
It process dictionary type `examples` as a parameter.
It is used in `train_dataset = train_dataset.map(tokenize, batched=True)`
No error is raised.

And then, I found some unnecessary keys and values in `examples` so I added `input_columns` argument to `map` function to select keys and values.
It gives me an error saying
```
TypeError: tokenize() takes 1 positional argument but 3 were given.
```

The code below matters.
https://github.com/huggingface/datasets/blob/406b2212263c0d33f267e35b917f410ff6b3bc00/src/datasets/iterable_dataset.py#L687

For example, `inputs = {"a":1, "b":2, "c":3}`.
If `self.input_coluns` is `None`,
`inputs` is a dictionary type variable and `function_args` becomes a `list` of a single `dict` variable.
`function_args` becomes `[{"a":1, "b":2, "c":3}]`

Otherwise, lets say `self.input_columns = ["a", "c"]`
`[inputs[col] for col in self.input_columns]` results in `[1, 3]`.
I think it should be `[{"a":1, "c":3}]`.

I want to ask if the resulting format is intended.
Maybe I can modify `tokenize()` to have 2 parameters in this case instead of having 1 dictionary.
But this is confusing to me.
Or it should be fixed as `[{col:inputs[col] for col in self.input_columns}]`

### Steps to reproduce the bug

Run `map` function of `IterableDataset` with `input_columns` argument.

### Expected behavior

`function_args` looks better to have same format.
I think it should be `[{"a":1, "c":3}]`.

### Environment info

dataset version: 2.12
python: 3.8

Comments:
Yes, this is intended.
Issue: Dataset string representation enhancement
State: open
Author: Ganryuu

my attempt at #6010  
not sure if this is the right way to go about it, I will wait for your feedback 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6016). All of your documentation changes will be reflected on that endpoint.

It we could have something similar to Polars, that would be great.

This is what Polars outputs: 
* `__repr__`/`__str__` :
```
shape: (67_349, 3)
┌───────┬───────────────────────────────────┬───────┐
│ idx   ┆ sentence                          ┆ label │
│ ---   ┆ ---                               ┆ ---   │
│ i32   ┆ str                               ┆ i64   │
╞═══════╪═══════════════════════════════════╪═══════╡
│ 0     ┆ hide new secretions from the par… ┆ 0     │
│ 1     ┆ contains no wit , only labored g… ┆ 0     │
│ 2     ┆ that loves its characters and co… ┆ 1     │
│ 3     ┆ remains utterly satisfied to rem… ┆ 0     │
│ …     ┆ …                                 ┆ …     │
│ 67345 ┆ anguish , anger and frustration   ┆ 0     │
│ 67346 ┆ at achieving the modest , crowd-… ┆ 1     │
│ 67347 ┆ a patient viewer                  ┆ 1     │
│ 67348 ┆ this new jangle of noise , mayhe… ┆ 0     │
└───────┴───────────────────────────────────┴───────┘
```

* `_repr_html_`:
<img width="251" alt="Screenshot 2023-07-12 at 18 25 58" src="https://github.com/huggingface/datasets/assets/47462742/5d04519d-f302-4411-9fbc-7445bdf53b23">
Issue: [FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage
State: open
Author: NightMachinery

### Feature request

Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns. 

### Motivation

This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.

### Your contribution

_

Comments:
You can use the `remove_columns` parameter in `map` to avoid duplicating the columns (and save disk space) and then concatenate the original dataset with the map result:
```python
from datasets import concatenate_datasets
# dummy example
ds_new = ds.map(lambda x: {"new_col": x["col"] + 2}, remove_columns=ds.column_names)
ds_combined = concatenate_datasets([ds, ds_new], axis=1)
```

Doing this automatically is hard to implement efficiently unless we know ahead of time which existing columns will be modified by a `map` transform. We have this info when `input_columns` are specified, so I think this is the only case we can optimize.
Issue: Improve `Dataset`'s string representation
State: open
Author: mariosasko

Currently, `Dataset.__repr__` outputs a dataset's column names and the number of rows. We could improve it by printing its features and the first few rows.

We should also implement `_repr_html_` to have a rich HTML representation in notebooks/Streamlit.

Comments:
I want to take a shot at this if possible 

Yes, feel free to work on this.

You can check the PyArrow Table `__repr__`  and Polars DataFrame `__repr__`/`_repr_html_` implementations for some pointers/ideas.

@mariosasko are there any other similar issues that I could work on? I see this has been already solved.
Issue: Dataset.from_generator consistently freezes at ~1000 rows
State: closed
Author: andreemic

### Describe the bug

Whenever I try to create a dataset which contains images using `Dataset.from_generator`, it freezes around 996 rows. I suppose it has something to do with memory consumption, but there's more memory available. I

Somehow it worked a few times but mostly this makes the datasets library much more cumbersome to work with because generators are the easiest way to turn an existing dataset into a Hugging Face dataset.

I've let it run in the frozen state for way longer than it can possibly take to load the actual dataset.

Let me know if you have ideas how to resolve it!

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

def gen():
    for row in range(10000):
        yield {"i": np.random.rand(512, 512, 3)}
        
Dataset.from_generator(gen)
# -> 90% of the time gets stuck around 1000 rows
```

### Expected behavior

Should continue and go through all the examples yielded by the generator, or at least throw an error or somehow communicate what's going on.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 12.0.1
- Pandas version: 1.5.1


Comments:
By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).

Our support for multi-dim arrays could be better, and we plan to improve it as part of https://github.com/huggingface/datasets/issues/5272.

> By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).
> 
> Our support for multi-dim arrays could be better, and we plan to improve it as part of #5272.

Thanks for the explanation! The Image array was just for demonstration, I use PIL Images in practice. Does that make a difference? What's the best approach for a dataset with PIL Images as rows?

It's best to use the `datasets.Image()` feature type for PIL images (to save space) :)
Issue: Add KLUE-MRC metrics
State: closed
Author: ingyuseong

## Metrics for KLUE-MRC (Korean Language Understanding Evaluation — Machine Reading Comprehension)

Adding metrics for [KLUE-MRC](https://huggingface.co/datasets/klue).
KLUE-MRC is very similar to SQuAD 2.0 but has a slightly different format which is why I added metrics for KLUE-MRC.

Specifically, in the case of [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness), it leverages the scoring script of SQuAD to evaluate SQuAD 2.0 and KorQuAD. But the script isn't suitable for KLUE-MRC because KLUE-MRC is a bit different from SQuAD 2.0. And this is why I added the scoring script for KLUE-MRC.

- [x] All tests passed
- [x] Added a metric card (referred the metric card of SQuAD 2.0)
- [x] Compatibility test with [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) passed

### References
- [KLUE: Korean Language Understanding Evaluation](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/98dce83da57b0395e163467c9dae521b-Paper-round2.pdf)
- [KLUE on Hugging Face Datasets](https://huggingface.co/datasets/klue)
- #2416

Comments:
The metrics API in `datasets` is deprecated as of version 2.0, and `evaulate` is our new library for metrics. You can add a new metric to it by following [these steps](https://huggingface.co/docs/evaluate/creating_and_sharing).
Issue: Getting a 409 error while loading xglue dataset
State: closed
Author: Praful932

### Describe the bug

Unable to load xglue dataset

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.load_dataset("xglue", "ntg")
```

> ConnectionError: Couldn't reach https://xglue.blob.core.windows.net/xglue/xglue_full_dataset.tar.gz (error 409)

### Expected behavior

Expected the dataset to load

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Thanks for reporting, @Praful932.

Let's continue the conversation on the Hub: https://huggingface.co/datasets/xglue/discussions/5
Issue: The current implementation has a potential bug in the sort method
State: closed
Author: wangyuxinwhy

### Describe the bug

In the sort method，here's a piece of code

```python
# column_names: Union[str, Sequence_[str]]

# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    column_names = [column_names]
```

I get an error when I pass in a tuple based on the column_names type annotation, it will raise an errror.As in the example below, while the type annotation implies that a tuple can be passed.

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

Of course, after I modified the tuple into a list, everything worked fine

Change the code to the following so there will be no problem

```python
# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    if isinstance(column_names, str):
        column_names = [column_names]
    else:
        column_names = list(column_names)
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

### Expected behavior

Passing tuple into column_names should be equivalent to passing list

### Environment info

- `datasets` version: 2.13.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2

Comments:
Thanks for reporting, @wangyuxinwhy.
Issue: ValueError: Table schema does not match schema used to create file
State: closed
Author: exs-avianello

### Describe the bug

Saving a dataset as parquet fails with a `ValueError: Table schema does not match schema used to create file`  if the dataset was obtained out of a `.select_columns()` call with columns selected out of order.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_parquet("demo.parquet")
```

```shell
>>>
ValueError: Table schema does not match schema used to create file: 
table:
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53 vs. 
file:
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53
```

--- 

I think this is because after the `.select_columns()` call with out of order columns, the output dataset features' schema ends up being out of sync with the schema of the arrow table backing it. 

```python
ds.features.arrow_schema
>>>
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53

ds.data.schema
>>>
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53
```


So when we call `.to_parquet()`, the call behind the scenes to `datasets.io.parquet.ParquetDatasetWriter(...).write()` which initialises the backend `pyarrow.parquet.ParquetWriter` with `schema = self.dataset.features.arrow_schema` triggers `pyarrow` on write when [it checks](https://github.com/apache/arrow/blob/11b140a734a516e436adaddaeb35d23f30dcce44/python/pyarrow/parquet/core.py#L1086-L1090) that the `ParquetWriter` schema matches the schema of the table being written 🙌 

https://github.com/huggingface/datasets/blob/6ed837325cb539a5deb99129e5ad181d0269e050/src/datasets/io/parquet.py#L139-L141


### Expected behavior

The dataset gets successfully saved as parquet. 

*In the same way as it does if saving it as csv:

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_csv("demo.csv")
```

### Environment info

`python==3.11`
`datasets==2.13.1`


Comments:
We'll do a new release of `datasets` soon to make the fix available :)

In the meantime you can use `datasets` from source (main)

Thank you very much @lhoestq ! 🚀
Issue: `map` with any joblib backend
State: open
Author: lhoestq

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).

Comments:
Issue: Set a rule on the config and split names
State: open
Author: severo

> should we actually allow characters like spaces? maybe it's better to add validation for whitespace symbols and directly in datasets and raise

https://github.com/huggingface/datasets-server/issues/853


Comments:
in this case we need to decide what to do with the existing datasets with white space characters (there shouldn't be a lot of them I think)

I imagine that we should stop supporting them, and help the user fix them?

See a report where the datasets server fails: https://huggingface.co/datasets/poloclub/diffusiondb/discussions/2#6374ff55b93cbdf65675f564

The config name is `random_10k [2m]`!