Kokolipa/github-issues huggingface dataset:
Issue: Token Alignment for input and output data over train and test batch/dataset.
State: open

`data`
> DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 4500
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 500
    })
})

**# input (in-correct sentence)**
`data['train'][0]['input']`
**>>** 'We are meet sunday 10am12pmET in Crown Heights Brooklyn New York'
**# output (correct sentence)**
`data['train'][0]['output']`
**>>** 'We meet Sundays 10am-12pmET in Crown Heights, Brooklyn, New York.'

**I Want to align the output tokens with input**

```
`# tokenize both inputs and targets
def tokenize_fn(batch):
    # tokenize the input sequence first
    # this populates input_ids, attention_mask, etc.

    tokenized_inputs = tokenizer(
    batch['input']
    )
    
    labels_batch = tokenizer.tokenize(batch['output']) # original targets

    aligned_labels_batch = []
    for i, labels in enumerate(labels_batch):
        word_ids = tokenized_inputs[i].word_ids()
        aligned_labels_batch.append(align_targets(labels, word_ids))  # align_targets is another user defined function which is been called here

    # recall: the 'target' must be stored in key called 'labels'
    tokenized_inputs['labels'] = aligned_labels_batch

    return tokenized_inputs`
```
```
data.map(
  tokenize_fn,
  batched=True,
  remove_columns=data['train'].column_names,
)
```

When this user defined function is mapped to every records of train and test batch am getting following error:

**1.** **raise DatasetTransformationNotAllowedError(
   3457         "Using `.map` in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."**

**2.** **TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]**
Issue: Use work-stealing algorithm when parallel computing
State: open
Labels: name: enhancement

### Feature request

when i used Dataset.map api to process data concurrently, i found that
 it gets slower and slower as it gets closer to completion. Then i read the source code of arrow_dataset.py and found that it shard the dataset and use multiprocessing pool to execute each shard.It may cause the slowest task to drag out the entire program's execution time,especially when processing huge dataset.

### Motivation

using work-stealing algorithm instead of sharding and parallel computing to optimize performance. 

### Your contribution

just an idea.

Alternatively we could set the number of shards to be a factor than the number of processes (current they're equal) - this way it will be less likely to end up with a shard that is significantly slower than all the other ones.
Issue: `Dataset.to_tf_dataset` fails when strings cannot be encoded as `np.bytes_`
State: closed

### Describe the bug

When loading any dataset that contains a column with strings that are not ASCII-compatible, looping over those records raises the following exception e.g. for `é` character `UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)`.

### Steps to reproduce the bug

Running the following script will eventually fail, when reaching to the batch that contains non-ASCII compatible strings.

```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train")
tfds = ds.to_tf_dataset(batch_size=16)

for batch in tfds:
  print(batch)
>>> UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)
```

### Expected behavior

The following script to run properly, making sure that the strings are either `numpy.unicode_` or `numpy.string` instead of `numpy.bytes_` since some characters are not ASCII compatible and that would lead to an issue when applying the `map`.

```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train")
tfds = ds.to_tf_dataset(batch_size=16)

for batch in tfds:
  print(batch)
```

### Environment info

- `datasets` version: 2.12.1.dev0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

May eventually be solved in #5883 

#self-assign
Issue: load_dataset from s3 file system through streaming can't not iterate data 
State: open

### Describe the bug

I have a JSON file in my s3 file system(minio), I can use load_dataset to get the file link, but I can't iterate it
<img width="816" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/cc0778d3-36f3-45b5-ac68-4e7c664c2ed0">
<img width="1144" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/76872af3-8b3c-42ff-9f55-528c920a7af1">

we can change 4 lines to fix this bug, you can check whether it is ok for us.
<img width="941" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/5a22155a-ece7-496c-8506-047e5c235cd3">

### Steps to reproduce the bug

1. storage a file in you s3 file system
2. use load_dataset to read it through streaming
3. iterate it

### Expected behavior

can iterate it successfully

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1


This sounds related to #5281.

Can you try passing `storage_options=s3_client.storage_options` instead passing it to `use_auth_token=` ?

I tried `storage_options` before, but it doesn't work, I checked our source code and I found that we even didn't pass this parameter to the following process. if I use `storage_options` instead of `use_auth_token`, then I also need to change another place of the code. the last line of `streaming_download_manager.py`. our code only passes the `use_auth_token` to the following handler, but does nothing to the `storage_options`
<img width="1050" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/5be90933-3331-4ecf-9e11-34f9852d8f92">


Cloud storage support is still experimental indeed and you can expect some bugs.

I think we need to pass the storage options anywhere use_auth_token is passed in indeed. Let me know if you'd be interested in contributing a fix !

Oh, that's great, I really like to fix it. because datasets is really useful and most of our projects need to use it, but we can store our data on the internet due to security reasons. fix it not only make our own work more efficient but also can benefit others who use it.
Issue: Request for text deduplication feature
State: open
Labels: name: enhancement

### Feature request

It would be great if there would be support for high performance, highly scalable text deduplication algorithms as part of the datasets library.

### Motivation

Motivated by this blog post https://huggingface.co/blog/dedup and this library https://github.com/google-research/deduplicate-text-datasets, but slightly frustrated by how its not very easy to work with these tools I am proposing this feature.

### Your contribution

I would be happy to contribute to the development effort of this feature. would love to collaborate with others in the development effort. 

The "exact match" deduplication will be possible when we resolve https://github.com/huggingface/datasets/issues/2514 (first, https://github.com/apache/arrow/issues/30950 needs to be addressed on the Arrow side). In the meantime, you can use Polars or DuckDB (e.g., via [datasets-sql](https://github.com/mariosasko/datasets_sql)).

Fuzzy deduplication is out-of-scope for now ([splink](https://github.com/moj-analytical-services/splink) is probably the best tool for it).

This library can be an intermediate solution : https://github.com/ChenghaoMou/text-dedup/tree/main

I have been using polars to remove duplicates but it would be nice to do it directly in pyarrow.

For example,

1. Read dataset with pyarrow
2. Use scan_pyarrow_dataset() with Polars to create a LazyFrame
3. Use sort and unique to remove duplicates based on a subset of columns
4. Convert to table and save data with ds.write_dataset()

There are times where that workflow makes perfect sense because I do additional transformations with Polars. Most of the time I am simply just reading dataset A and writing dataset B without duplicates though, and I wish I could use a pyarrow scanner or table directly. 

Hi
see this new release  from hf [datatrove](https://github.com/huggingface/datatrove)
DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality
Issue: Why split slicing doesn't behave like list slicing ?
State: closed
Labels: name: duplicate

### Describe the bug

If I want to get the first 10 samples of my dataset, I can do :

```
ds = datasets.load_dataset('mnist', split='train[:10]')
```

But if I exceed the number of samples in the dataset, an exception is raised : 

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

> ValueError: Requested slice [:999999999] incompatible with 60000 examples.

### Steps to reproduce the bug

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

### Expected behavior

I would expect it to behave like python lists (no exception raised, the whole list is kept) : 

```
d = list(range(1000))[:999999]
print(len(d))  # > 1000
```

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

A duplicate of https://github.com/huggingface/datasets/issues/1774
Issue: Using as_dataset on a "parquet" builder  
State: closed

### Describe the bug

I used a custom builder to ``download_and_prepare`` a dataset. The first (very minor) issue is that the doc seems to suggest ``download_and_prepare`` will return the dataset, while it does not ([builder.py](https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L718-L738)).
```
        >>> from datasets import load_dataset_builder
        >>> builder = load_dataset_builder("rotten_tomatoes")
        >>> ds = builder.download_and_prepare("./output_dir", file_format="parquet")
```

The main issue I am facing is loading the dataset from those parquet files. I used the `as_dataset` method suggested by the doc, however it returns:
`
FileNotFoundError: [Errno 2] Failed to open local file 'output_dir/__main__-train-00000-of-00245.arrow'. Detail:
[errno 2] No such file or directory.
` 

### Steps to reproduce the bug

1. Create a custom builder of some sort: `builder = CustomBuilder()`.
2. Run `download_and_prepare` with the parquet format: `builder.download_and_prepare("./output_dir", file_format="parquet")`.
3. Run `dataset = builder.as_dataset()`.  

### Expected behavior

I guess I'd expect `as_dataset` to generate the dataset in arrow format if it has to, or to suggest an alternative way to load the dataset (I've also tried other methods with `load_dataset` to no avail, probably due to misunderstandings on my part). 

### Environment info

```
- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.14.1
- PyArrow version: 8.0.0
- Pandas version: 1.5.3
```

Hi! You can refer to [this doc](https://huggingface.co/docs/datasets/filesystems#load-and-save-your-datasets-using-your-cloud-storage-filesystem) to see the intended usage (basically, it skips the Arrow -> Parquet conversion step in `ds = load_dataset(...); ds.to_parquet("path/to/parquet")`) and allows writing Parquet to remote storage unlike `to_parquet`).

> I guess I'd expect as_dataset to generate the dataset in arrow format if it has to, or to suggest an alternative way to load the dataset (I've also tried other methods with load_dataset to no avail, probably due to misunderstandings on my part).

`as_dataset` does not work with `file_format="parquet"` files as Parquet files cannot be memory-mapped, so I think we should just raise an error in that case.
Issue: Allow setting the environment variable for the lock file path
State: open
Labels: name: enhancement

### Feature request

Add an environment variable to replace the default lock file path.

### Motivation

Usually, dataset path is a read-only path while the lock file needs to be modified each time. It would be convenient if the path can be reset individually.

### Your contribution

```/src/datasets/utils/filelock.py
class UnixFileLock(BaseFileLock):
    def __init__(self, lock_file, timeout=-1, max_filename_length=None):
        #-------------------
        if os.getenv('DS_TMP_PATH'):
            file_name = str(lock_file).split('/')[-1]
            dataset_tmp_path = os.getenv('DS_TMP_PATH')
            lock_file = os.path.join(dataset_tmp_path, file_name)
        #-------------------
        max_filename_length = os.statvfs(os.path.dirname(lock_file)).f_namemax
        super().__init__(lock_file, timeout=timeout, max_filename_length=max_filename_length)
```
A simple demo is as upper. Thanks.
Issue: data configuration hash suffix depends on uncanonicalized data_dir
State: closed
Labels: name: good first issue

### Describe the bug

I am working with the `recipe_nlg` dataset, which requires manual download. Once it's downloaded, I've noticed that the hash in the custom data configuration is different if I add a trailing `/` to my `data_dir`. It took me a while to notice that the hashes were different, and to understand that that was the cause of my dataset being processed anew instead of the cached version being used.

### Steps to reproduce the bug

1. Follow the steps to manually download the `recipe_nlg` dataset to `/data/recipenlg`.
2. Load it using `load_dataset`, once without a trailing slash and once with one:

    ```python
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg")
    Using custom data configuration default-082278caeea85765
    Downloading and preparing dataset recipe_nlg/default to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Dataset recipe_nlg downloaded and prepared to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74. Subsequent calls will reuse this data.
    100%|███████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.10s/it]
    DatasetDict({
        train: Dataset({
            features: ['id', 'title', 'ingredients', 'directions', 'link', 'source', 'ner'],
            num_rows: 2231142
        })
    })
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg/")
    Using custom data configuration default-83e87680785d0493
    Downloading and preparing dataset recipe_nlg/default to /home/user/.cache/huggingface/datasets/recipe_nlg/default-83e87680785d0493/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Generating train split:   1%|          | 12701/2231142 [00:04<13:15, 2790.25 examples/s
    ^C
    ```

3. Observe that the hash suffix in the custom data configuration changes due to the altered string.

### Expected behavior

I think I would expect the hash to remain constant if it actually points to the same location on disk. I would expect the use of `os.path.normpath` to canonicalize the paths.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-147-generic-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

It could even use `os.path.realpath` to resolve symlinks.

Indeed, it makes sense to normalize `data_dir`. Feel free to submit a PR (this can be "fixed" [here](https://github.com/huggingface/datasets/blob/89f775226321ba94e5bf4670a323c0fb44f5f65c/src/datasets/builder.py#L173))

#self-assign
Issue: Is it possible to change a cached file and 're-cache' it instead of re-generating?
State: closed
Labels: name: enhancement

### Feature request

Hi,
I have a huge cached file using `map`(over 500GB), and I want to change an attribution of each element, is there possible to do it using some method instead of re-generating, because `map` takes over 24 hours

### Motivation

For large datasets, I think it is very important because we always face the problem which is changing something in the original cache without re-generating it.

### Your contribution

For now, I can't help, sorry.

Arrow files/primitives (tables and arrays) are immutable, so re-generating them is the only option, I'm afraid.

> 

Got it, thanks for your reply
Issue: Issue with Sequence features
State: closed

### Describe the bug

Sequences features  sometimes causes errors when the specified length is not -1

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Features, ClassLabel, Sequence, Value, Dataset
feats = Features(**{'target': ClassLabel(names=[0, 1]),'x': Sequence(feature=Value(dtype='float64',id=None), length=2, id=None)})
Dataset.from_dict({"target": np.ones(2000).astype(int), "x": np.random.rand(2000,2)},features = feats).flatten_indices()
```
Throws:
```
  TypeError: Couldn't cast array of type
  fixed_size_list<item: double>[2]
  to
  Sequence(feature=Value(dtype='float64', id=None), length=2, id=None)
```
The same code works without any issues when `length = -1`

EDIT: The error seems to happen only when the length of the dataset is bigger than 1000 for some reason
### Expected behavior

No exception

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1

Thanks for reporting! I've opened a PR with a fix.
Issue: IndexError: list index out of range with data hosted on Zenodo
State: open
Labels: name: bug

The dataset viewer sometimes raises an `IndexError`:
```
IndexError: list index out of range
```
See:
- huggingface/datasets-server#1151
  - https://huggingface.co/datasets/reddit/discussions/5
- huggingface/datasets-server#1118
  - https://huggingface.co/datasets/krr-oxford/OntoLAMA/discussions/1
- https://huggingface.co/datasets/hyperpartisan_news_detection/discussions/3
- https://huggingface.co/datasets/um005/discussions/2
- https://huggingface.co/datasets/tapaco/discussions/2
- https://huggingface.co/datasets/common_language/discussions/3
- https://huggingface.co/datasets/pass/discussions/1

After investigation:
- This happens with data files hosted on Zenodo
- Indeed, there is an underlying 429 HTTP error: Too Many Requests

Note that some time ago, it also happened with data files hosted on Google Drive. See:
  - #4581
  - #4580 

The reason then was that there was a 403 HTTP error: Forbidden



This error is also raised when data is hosted on Google Drive:
- https://huggingface.co/datasets/docred/discussions/5
- https://huggingface.co/datasets/linnaeus/discussions/3
- https://huggingface.co/datasets/poleval2019_mt/discussions/3
- https://huggingface.co/datasets/reddit_tifu/discussions/2
- https://huggingface.co/datasets/species_800/discussions/3
- https://huggingface.co/datasets/wiki_lingua/discussions/1
- https://huggingface.co/datasets/yoruba_text_c3/discussions/1
Issue: Throw an error when dataset improperly indexed
State: closed

### Describe the bug

Pandas-style subset indexing on dataset does not throw an error, when maybe it should. Instead returns the first instance of the dataset regardless of index condition.

### Steps to reproduce the bug

Steps to reproduce the behavior:

1. `squad = datasets.load_dataset("squad_v2", split="validation")`
2. `item = squad[squad['question'] ==  "Who was the Norse leader?"]`
or `it = squad[squad['id'] == '56ddde6b9a695914005b962b']`
3.  returns the first item in the dataset, which does not satisfy the above conditions:

`{'id': '56ddde6b9a695914005b9628', 'title': 'Normans', 'context': 'The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.', 'question': 'In what country is Normandy located?', 'answers': {'text': ['France', 'France', 'France', 'France'], 'answer_start': [159, 159, 159, 159]}}`

### Expected behavior

Should either throw an error message, or return the dataset item that satisfies the condition.

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Thanks for reporting, @sarahwie.

Please note that in `datasets` we do not have vectorized operation like `pandas`. Therefore, your equality comparisons above are `False`:
- For example: `squad['question']` returns a `list`, and this list is not equal to `"Who was the Norse leader?"`

The `False` value is equivalent to `0` when indexing a dataset, thus the reason why you get the first element (with index 0): 
- For example: `squad[False]` is equivalent to `squad[0]`

Maybe we should an exception instead of assuming that `False` is equivalent to `0` (and `True` is equivalent to `1`) in the context of indexing.
Issue: Adding chemistry dataset/models in huggingface
State: closed
Labels: name: enhancement

### Feature request

Huggingface is really amazing platform for open science.

In addition to computer vision, video and NLP, would it be of interest to add chemistry/materials science dataset/models in Huggingface? Or, if its already done, can you provide some pointers.

We have been working on a comprehensive benchmark on this topic: [JARVIS-Leaderboard](https://pages.nist.gov/jarvis_leaderboard/) and I am wondering if we could contribute/integrate this project as a part of huggingface. 

### Motivation

Similar to the main stream AI field, there is need of large scale benchmarks/models/infrastructure for chemistry/materials data.

### Your contribution

We can start adding datasets as our [benchmarks](https://github.com/usnistgov/jarvis_leaderboard/tree/main/jarvis_leaderboard/benchmarks) should be easily convertible to the dataset format.

Hi! 

This would be a nice addition to the Hub! You can find the existing chemistry datasets/models on the Hub (using the `chemistry` tag) [here](https://huggingface.co/search/full-text?q=chemistry&type=model&type=dataset).

Feel free to ping us here on the Hub if you need help adding the datasets.
Issue: Error loading natural_questions
State: closed

### Describe the bug

When try to load natural_questions through datasets == 2.12.0 with python == 3.8.9:

```python
import datasets
datasets.load_dataset('natural_questions',beam_runner='DirectRunner')
```

It failed with following info:

`pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs`

### Steps to reproduce the bug

In python console:

```python
import datasets
datasets.load_dataset('natural_questions',beam_runner='DirectRunner')
```

Then the trace is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 2019, in _download_and_prepare
    num_examples, num_bytes = beam_writer.finalize(metrics.query(m_filter))
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/arrow_writer.py", line 694, in finalize
    shard_num_bytes, _ = parquet_to_arrow(source, destination)
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/arrow_writer.py", line 737, in parquet_to_arrow
    for record_batch in parquet_file.iter_batches():
  File "pyarrow/_parquet.pyx", line 1323, in iter_batches
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs
```

### Expected behavior

load natural_question questions

### Environment info

```
- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.9
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.1
```

Hi! You can avoid this error by using the preprocessed version:
```python
import datasets
ds = datasets.load_dataset('natural_questions')
```

PS: Once we finish https://github.com/huggingface/datasets/pull/5364, this error will no longer be a problem.

> Hi! You can avoid this error by using the preprocessed version:
> 
> ```python
> import datasets
> ds = datasets.load_dataset('natural_questions')
> ```
> 
> PS: Once we finish #5364, this error will no longer be a problem.

Thanks, wish #5364  finish early
Issue: Can not load audiofolder dataset on kaggle
State: closed

### Describe the bug
It's crash log:
FileNotFoundError: Couldn't find a dataset script at /kaggle/working/audiofolder/audiofolder.py or any data file in the same directory. Couldn't find 'audiofolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/audiofolder/audiofolder.py


### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/93691919/a2829d27-d15c-4acc-86fb-d1987c760468)
common_voice = load_dataset("audiofolder", data_dir="/kaggle/working/data")

### Expected behavior

load dataset without error. It works ok on colab, but on kaggle it happends.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment (and restart the env for the update to take effect) to resolve the issue.

> Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment to resolve the issue.

I don't think it is a problem of the version. It runs ok on colab or local machine. Only on kaggle will has this bug.

Based on your dataset info, the installed version is `2.1.0`, which does not include `audiofolder`.

By default, Kaggle preinstalls `datasets` into a new env, but the version it installs is outdated and does not contain newer features such as `audiofolder`
Issue: CSV datasets should only read the CSV data files in the repo
State: closed
Labels: name: bug

When a no-script dataset has many CSV files and a JPG file, the library infers to use the Csv builder, but tries to read as CSV all files in the repo, also the JPG file.

I think the Csv builder should filter out non-CSV files when reading.

An analogue solution should be implemented for other packaged builders.

Related to:
- https://huggingface.co/datasets/abidlabs/img2text/discussions/1
- https://github.com/gradio-app/gradio/pull/3973#issuecomment-1545409061

CC: @abidlabs @severo
Issue: Error message not clear in interleaving datasets
State: closed

### System Info

standard env

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm trying to interleave 'sciq', 'wiki' and the 'pile-enron' dataset. I think the error I made was that I loaded the train split of one, but for the other but the error is not too helpful- 

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/home/suryahari/Vornoi/save_model_ops.py](https://vscode-remote+ssh-002dremote-002bthomsonlab-002d2-002ejamesgornet-002ecom.vscode-resource.vscode-cdn.net/home/suryahari/Vornoi/save_model_ops.py) in line 3
      [41](file:///home/suryahari/Vornoi/save_model_ops.py?line=40) # %%
----> [43](file:///home/suryahari/Vornoi/save_model_ops.py?line=42) dataset = interleave_datasets(datasets, stopping_strategy="all_exhausted")

File [~/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py:124](https://vscode-remote+ssh-002dremote-002bthomsonlab-002d2-002ejamesgornet-002ecom.vscode-resource.vscode-cdn.net/home/suryahari/~/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py:124), in interleave_datasets(datasets, probabilities, seed, info, split, stopping_strategy)
    [122](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=121) for dataset in datasets[1:]:
    [123](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=122)     if (map_style and not isinstance(dataset, Dataset)) or (iterable and not isinstance(dataset, IterableDataset)):
--> [124](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=123)         raise ValueError(
    [125](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=124)             f"Unable to interleave a {type(datasets[0])} with a {type(dataset)}. Expected a list of Dataset objects or a list of IterableDataset objects."
    [126](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=125)         )
    [127](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=126) if stopping_strategy not in ["first_exhausted", "all_exhausted"]:
    [128](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=127)     raise ValueError(f"{stopping_strategy} is not supported. Please enter a valid stopping_strategy.")

ValueError: Unable to interleave a  with a . Expected a list of Dataset objects or a list of IterableDataset objects.
```

### Expected behavior

the error message should hopefully be more clear
Issue: load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!
State: closed

### Describe the bug

Running

```
import datasets
ds = datasets.load_dataset('bigcode/the-stack-dedup', streaming=True)
```

takes about 2.5 minutes! 

I would expect this to be near instantaneous. With other datasets, the runtime is one or two seconds.


### Environment info

- `datasets` version: 2.11.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0

This is due to the slow resolution of the data files: https://github.com/huggingface/datasets/issues/5537.

We plan to switch to `huggingface_hub`'s `HfFileSystem` soon to make the resolution faster (will be up to 20x faster once we merge https://github.com/huggingface/huggingface_hub/pull/1443)



You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

> You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

That's unrelated to the problem discussed in this issue. 

> > You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
> 
> That's unrelated to the problem discussed in this issue.

Sorry, I misunderstood it.

Closing this issue as it has been addressed in `huggingface_hub`!

(This now takes 25s to execute on my machine.)

Thanks for the improvements! 🎉🎉

25 seconds is better but still about 2500x slower than this _should_ be! Loading a tiny 1-2KB metadata file is all that would be necessary with a better design.

Once we merge https://github.com/huggingface/huggingface_hub/pull/2103, this should only take a few seconds. 

For the 2500x speed-up (without metadata files with pre-cached results), we wouldn't even be allowed to use `os.path` functions or `requests`/`aiohttp` for HTTP requests, so I don't think this is feasible for us as it would make the code unreadable.

The HF Datasets Hub is (almost) platform-agnostic, so you are free to implement your own library (in a faster language than Python) to achieve this kind of performance, and we would be happy to support it 🙂.
Issue: load model error.
State: closed

### Describe the bug

I had trained one model use deepspeed, when I load the final load I get the follow error:
OSError: Can't load tokenizer for '/XXX/DeepSpeedExamples/applications/DeepSpeed-Chat/output/step3-models/1.3b/actor'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/home/fm001/hzl/Project/DeepSpeedExamples/applications/DeepSpeed-Chat/output/step3-models/1.3b/actor' is the correct path to a directory containing all relevant files for a BloomTokenizerFast tokenizer.


my load code is : python chat.py --path /XXX/DeepSpeedExamples/applications/DeepSpeed-Chat/output/step3-models/1.3b/actor/



### Steps to reproduce the bug

。。。

### Expected behavior

。。。

### Environment info

。。。

Please report this in the `transformers` repo, as it's not related to `datasets`
Issue: Remove columns in interable dataset
State: closed

### Feature request

Right now, remove_columns() produces a NotImplementedError for iterable style datasets

### Motivation

It would be great to have the same functionality irrespective of whether one is using an iterable or a map-style dataset

### Your contribution

hope and courage.

Transferring this issue as it's related to the 🤗 Datasets library 

Hi @surya-narayanan! Could you provide some code snippet?

This method has been recently added to the `IterableDataset`, so you need to update the `datasets`' installation (`pip install -U datasets`) to use it.
Issue: Can't add iterable datasets to a Dataset Dict.
State: closed

### System Info

standard env

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Get the following error:

TypeError: Values in `DatasetDict` should be of type `Dataset` but got type '<class 'datasets.iterable_dataset.IterableDataset'>'


### Expected behavior

should be able to add iterable datasets to a dataset dict

Transferring as this is relating to the 🤗 Datasets library

You need to use `IterableDatasetDict` instead of `DatasetDict` for iterable datasets.
Issue: Make models/functions optimized with `torch.compile` hashable
State: closed
Labels: name: enhancement

As reported in https://github.com/huggingface/datasets/issues/5819, hashing functions/transforms that reference a model, or a function, optimized with `torch.compile` currently fails due to them not being picklable (the concrete error can be found in the linked issue).

The solutions to consider:
1. hashing/pickling the original, uncompiled version of a compiled model/function (attributes `_orig_mod`/`_torchdynamo_orig_callable`) (less precise than the 2nd option as it ignores the other params of `torch.compute`)
2. wait for https://github.com/pytorch/pytorch/issues/101107 to be resolved
Issue: Is uint8 supported?
State: closed

### Describe the bug

I expect the dataset to store the data in the `uint8` data type, but it's returning `int64` instead.
While I've found that `datasets` doesn't yet support float16 (https://github.com/huggingface/datasets/issues/4981), I'm wondering if this is the case for other data types as well.
Is there a way to store vector data as `uint8` and then upload it to the hub?

### Steps to reproduce the bug

```python
from datasets import Features, Dataset, Sequence, Value
import numpy as np

dataset = Dataset.from_dict(
    {"vector": [np.array([0, 1, 2], dtype=np.uint8)]}, features=Features({"vector": Sequence(Value("uint8"))})
).with_format("numpy")

print(dataset[0]["vector"].dtype)
```

### Expected behavior

Expected: `uint8`
Actual: `int64`

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-12.1-x86_64-i386-64bit
- Python version: 3.8.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Hi ! The numpy formatting detaults to int64 and float32 - but you can use uint8 using
```python
ds = ds.with_format("numpy", dtype=np.uint8)
```

Related to https://github.com/huggingface/datasets/issues/5517.

Thank you!
By setting `ds.with_format("numpy", dtype=np.uint8)`, the dataset returns the data in `uint8`.

However, `with_format` and `set_format` seem to cast the data on-the-fly.
I want to reduce the dataset size by using `uint8` instead of `int64` and I observe no difference between using `int64` and `uint8` for the vector.
Is there any way to actually store the data in `uint8` and save the disk space and the downloading time when loaded from the hub?


If the feature type is `Value("uint8")` then it's written an uint8 on disk using the uint8 Arrow dtype.

e.g.
```python
ds = Dataset.from_dict({"a": range(10)}, features=Features({"a": Value("uint8")}))
ds.data.nbytes
# 10
```

Oh, I understand now.
The data was stored in `uint8` from the beginning (when the dataset returns `int64`).

Thank you for your time!
My question is fully resolved.
Issue: (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
State: closed

### Describe the bug

M2 MBP can't run
```python
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Steps to reproduce the bug

1. Use M2 MBP
2. Python 3.10.10 from pyenv
3. Run 
```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Expected behavior

Be able to run normally

### Environment info

```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```
OSX: 13.2
CPU: M2


Can you paste the error stack trace?

That is weird. I can't reproduce it again after reboot.
```python
In [2]: import platform

In [3]: platform.platform()
Out[3]: 'macOS-13.2-arm64-arm-64bit'

In [4]: from datasets import load_dataset
   ...:
   ...: jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
Found cached dataset parquet (/Users/sarit/.cache/huggingface/datasets/nomic-ai___parquet/nomic-ai--gpt4all-j-prompt-generations-a3b62015e2e52043/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 63.25it/s]
```
Issue: Stream data concatenation issue
State: closed

### Describe the bug

I am not able to concatenate the augmentation of the stream data. I am using the latest version of dataset.

ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string', 
id=None), 'audio': {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'path': 
Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'transcript': Value(dtype='string', 
id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 
'path': Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either 
Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").

### Steps to reproduce the bug

dataset = load_dataset("tobiolatunji/afrispeech-200", "all", streaming=True).shuffle(seed=42)
dataset_cln = dataset.remove_columns(['speaker_id', 'path', 'age_group', 'gender', 'accent', 'domain', 'country', 'duration'])
dataset_cln = dataset_cln.cast_column("audio", Audio(sampling_rate=16000))
from audiomentations import AddGaussianNoise,Compose,Gain,OneOf,PitchShift,PolarityInversion,TimeStretch

augmentation = Compose([
    AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=0.2)
])

def augment_dataset(batch):
    audio = batch["audio"]
    audio["array"] = augmentation(audio["array"], sample_rate=audio["sampling_rate"])    
    return batch

augmented_dataset_cln = dataset_cln['train'].map(augment_dataset)
dataset_cln['train'] = interleave_datasets([dataset_cln['train'], augmented_dataset_cln])
dataset_cln['train'] = dataset_cln['train'].shuffle(seed=42)

### Expected behavior

I should be able to merge as sampling rate is same.

### Environment info

import datasets
import transformers
import accelerate
print(datasets.__version__)
print(transformers.__version__)
print(torch.__version__)
print(evaluate.__version__)
print(accelerate.__version__)

2.12.0
4.28.1
2.0.0
0.4.0
0.18.0

Hi! 

You can call `map` as follows to avoid the error:
```python
augmented_dataset_cln = dataset_cln['train'].map(augment_dataset, features=dataset_cln['train'].features)
```

Thanks it is solved

Hi! 
I have run into the same problem with you. Could you please let me know how you solve it? Thanks!
Issue: load json dataset interrupt when dtype cast problem occured
State: open

### Describe the bug

i have a json like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}, 
    ....
]
，which have several problematic rows data like row 2, then i load it with datasets.load_dataset('json', data_files=['xx.json'], split='train'), it will report like this:
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file 'C:\Users\gawinjunwu\Downloads\test\data\a.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Could not convert '2' with type str: tried to convert to int64
Traceback (most recent call last):
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "D:\Python3.9\lib\site-packages\datasets\packaged_modules\json\json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at C:\Users\gawinjunwu\Downloads\test\data\a.json.     

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\gawinjunwu\Downloads\test\scripts\a.py", line 4, in <module>
    ds = load_dataset('json', data_dir='data', split='train')
  File "D:\Python3.9\lib\site-packages\datasets\load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset.
Could datasets skip those problematic data row? 

### Steps to reproduce the bug

prepare a json file like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}
]
then use datasets.load_dataset('json', dir_files=['xxx.json']) to load the json file

### Expected behavior

skip the problematic data row and load row1 and row3

### Environment info

python3.9

Indeed the JSON dataset builder raises an error when it encounters an unexpected type.

There's an old PR open to add away to ignore such elements though, if it can help: https://github.com/huggingface/datasets/pull/2838
Issue: [2.12.0] DatasetDict.save_to_disk not saving to S3
State: closed

### Describe the bug

When trying to save a `DatasetDict` to a private S3 bucket using `save_to_disk`, the artifacts are instead saved locally, and not in the S3 bucket.

I have tried using the deprecated `fs` as well as the `storage_options` arguments and I get the same results.

### Steps to reproduce the bug

1. Create a DatsetDict `dataset`
2. Create a S3FileSystem object
`s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)`
3. Save using `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", storage_options=s3.storage_options)` or `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", fs=s3)`
4. Check the corresponding S3 bucket and verify nothing has been uploaded
5. Check the path at f"{s3_bucket}/{s3_dir}/{dataset_name}" and verify that files have been saved there

### Expected behavior

Artifacts are uploaded at the f"{s3_bucket}/{s3_dir}/{dataset_name}" S3 location.

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-x86_64-i386-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

Hi ! Can you try adding the `s3://` prefix ?
```python
f"s3://{s3_bucket}/{s3_dir}/{dataset_name}"
```

Ugh, yeah that was it. Thank you!

Hi @thejamesmarq, by any chance, did you use multiprocessing `num_proc > 1` when saving your dataset on the s3 bucket ? I'm struggling making it work in a multiprocessing setting while everything works fine with one processor.
Issue: Incomplete docstring for `BuilderConfig`
State: closed
Labels: name: good first issue

Hi guys !
I stumbled upon this docstring while working on a project.
Some of the attributes have missing descriptions.

https://github.com/huggingface/datasets/blob/bc5fef5b6d91f009e4101684adcb374df2c170f6/src/datasets/builder.py#L104-L117

Thanks for reporting! You are more than welcome to improve `BuilderConfig`'s docstring.

This class serves an identical purpose as `tensorflow_datasets`'s `BuilderConfig`, and its docstring is [here](https://github.com/tensorflow/datasets/blob/a95e38b5bb018312c3d3720619c2a8ef83ebf57f/tensorflow_datasets/core/dataset_builder.py#L81), so feel free to re-use parts of it.