jamesLeeeeeee/datasets-github-issues huggingface dataset:
Return the name of the currently loaded file in the load_dataset function.
Author: s-JoL | Pull request: no | State: open

### Feature request

Add an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

### Motivation

When training large language models, machine problems may interrupt the training process. In such cases, it is common to load a previously saved checkpoint to resume training. I would like to be able to obtain the names of the previously trained data shards, so that I can skip these parts of the data during continued training to avoid overfitting and redundant training time.

### Your contribution

I currently use a dataset in jsonl format, so I am primarily interested in the json format. I suggest adding the file name to the returned table here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.
Improve `Create a dataset` tutorial 
Author: polinaeterna | Pull request: no | State: open

Our [tutorial on how to create a dataset](https://huggingface.co/docs/datasets/create_dataset) is a bit misleading. 
1. In **Folder-based builders** section it says that we have two folder-based builders as standard builders, but we also have similar builders (that can be created from directory with data of required format) for `csv`, `json/jsonl`, `parquet` and `txt` files. We have info about these loaders in separate [guide for loading](https://huggingface.co/docs/datasets/loading#local-and-remote-files) but it's worth briefly mentioning them in the beginning tutorial because they are more common and for consistency. Would be helpful to add the link to the full guide.
2. **From local files** section lists methods for creating a dataset from in-memory data which are also described in [loading guide](https://huggingface.co/docs/datasets/loading#inmemory-data).  

Maybe we should actually rethink and restructure this tutorial somehow.
Files downloaded to cache do not respect umask
Author: albertvillanova | Pull request: no | State: closed

As reported by @stas00, files downloaded to the cache do not respect umask:
```bash
$ ls -l /path/to/cache/datasets/downloads/
-rw------- 1 uername    username    150M Apr 25 16:41 5e646c1d600f065adaeb134e536f6f2f296a6d804bd1f0e1fdcd20ee28c185c6 
```

Related to:
- #2065
Support parallelized downloading and processing in load_dataset with Spark
Author: es94129 | Pull request: no | State: open

### Feature request

When calling `load_dataset` for datasets that have multiple files, support using Spark to distribute the downloading and processing job to worker nodes when `cache_dir` is a cloud file system shared among nodes.

```python
load_dataset(..., use_spark=True)
```

### Motivation

Further speed up `dl_manager.download` and `_prepare_split` by distributing the workloads to worker nodes. 

### Your contribution

I can submit a PR to support this.
load_dataset is case sentitive?
Author: haonan-li | Pull request: no | State: open

### Describe the bug

load_dataset() function is case sensitive?

### Steps to reproduce the bug

The following two code, get totally different behavior.

1. load_dataset('mbzuai/bactrian-x','en')

2. load_dataset('MBZUAI/Bactrian-X','en')

### Expected behavior

Compare 1 and 2.
1 will download all 52 subsets, shell output:
```Downloading and preparing dataset json/MBZUAI--bactrian-X to xxx```
2 will only download single subset, shell output
```Downloading and preparing dataset bactrian-x/en to xxx```


### Environment info

Python 3.10.11
datasets Version: 2.11.0
CI ZeroDivisionError
Author: albertvillanova | Pull request: no | State: open

Sometimes when running our CI on Windows, we get a ZeroDivisionError:
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - ZeroDivisionError: float division by zero
```

See for example:
- https://github.com/huggingface/datasets/actions/runs/4809358266/jobs/8560513110
- https://github.com/huggingface/datasets/actions/runs/4798359836/jobs/8536573688

```
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

split = 'test', start_time = 1682516718.8236516, num_samples = 2, num_steps = 1

    def speed_metrics(split, start_time, num_samples=None, num_steps=None):
        """
        Measure and return speed performance metrics.
    
        This function requires a time snapshot `start_time` before the operation to be measured starts and this function
        should be run immediately after the operation to be measured has completed.
    
        Args:
        - split: name to prefix metric (like train, eval, test...)
        - start_time: operation start time
        - num_samples: number of samples processed
        """
        runtime = time.time() - start_time
        result = {f"{split}_runtime": round(runtime, 4)}
        if num_samples is not None:
>           samples_per_second = num_samples / runtime
E           ZeroDivisionError: float division by zero

C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\transformers\trainer_utils.py:354: ZeroDivisionError
```
IterableDataset.with_format("torch") not working
Author: jiangwangyi | Pull request: no | State: closed

### Describe the bug

After calling the with_format("torch") method on an IterableDataset instance, the data format is unchanged.

### Steps to reproduce the bug

```python
from datasets import IterableDataset
def gen():
    for i in range(4):
        yield {"a": [i] * 4}
dataset = IterableDataset.from_generator(gen).with_format("torch")
next(iter(dataset))
```

### Expected behavior

`{"a": torch.tensor([0, 0, 0, 0])}` is expected, but `{"a": [0, 0, 0, 0]}` is observed.

### Environment info

```bash
platform==ubuntu 22.04.01
python==3.10.9
datasets==2.11.0
```
TIFF/TIF support 
Author: sebasmos | Pull request: no | State: closed

### Feature request

I currently have a dataset (with tiff and json files) where I have to do this:

`wget path_to_data/images.zip && unzip images.zip`

`wget path_to_data/annotations.zip && unzip annotations.zip`
 
 Would it make sense a contribution that supports these type of files? 

### Motivation

instead of using `load_dataset` have to use wget as these files are not supported for annotations with JSON and images with TIFF files. 


Additionally to this, the PIL formatting from datasets does not read correctly the image channels with TIFF format, besides multichannel adaptation might be necessary as well (as my data e.g has more than 3 channels)

### Your contribution

1. Support TIFF images over multi channel format
2. Support JSON annotations
Allow to run CI on push to ci-branch
Author: albertvillanova | Pull request: yes | State: closed

This PR allows to run the CI on push to a branch named "ci-*", without needing to open a PR.
- This will allow to make CI tests without opening a PR, e.g., for future `huggingface-hub` releases, future dependency releases (like `fsspec`, `pandas`,...)

Note that to build the documentation, we already allow it on push to a branch named "doc-builder*".

See:
- #5788

CC: @Wauplin
Support streaming datasets that use jsonlines
Author: albertvillanova | Pull request: no | State: open

Extend support for streaming datasets that use `jsonlines.open`.

Currently, if `jsonlines` is installed, `datasets` raises a `FileNotFoundError`:
```
FileNotFoundError: [Errno 2] No such file or directory: 'https://...'
```

See:
- https://huggingface.co/datasets/masakhane/afriqa/discussions/1
Prepare tests for hfh 0.14
Author: Wauplin | Pull request: yes | State: closed

Related to the coming release of `huggingface_hub==0.14.0`. It will break some internal tests. The PR fixes these tests. Let's double-check the CI but I expect the fixed tests to be running fine with both `hfh<=0.13.4` and `hfh==0.14`. Worth case scenario, existing PRs will have to be rebased once this fix is merged.

See related [discussion](https://huggingface.slack.com/archives/C02V5EA0A95/p1682337463368609?thread_ts=1681994202.635609&cid=C02V5EA0A95) (private slack).

cc @lhoestq
Multiprocessing in a `filter` or `map` function with a Pytorch model
Author: HugoLaurencon | Pull request: no | State: closed

### Describe the bug

I am trying to use a Pytorch model loaded on CPUs with multiple processes with a `.map` or a `.filter` method.

Usually, when dealing with models that are non-pickable, creating a class such that the `map` function is the method `__call__`, and adding `reduce` helps to solve the problem.

However, here, the command hangs without throwing an error.


### Steps to reproduce the bug

```
from datasets import Dataset
import torch
from torch import nn
from torchvision import models
​
​
class FilterFunction:
    #__slots__ = ("path_model", "model")  # Doesn't change anything uncommented
    def __init__(self, path_model):
        self.path_model = path_model
        model = models.resnet50()
        model.fc = nn.Sequential(
            nn.Linear(2048, 512),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(512, 10),
            nn.LogSoftmax(dim=1)
        )
        model.load_state_dict(torch.load(path_model, map_location=torch.device("cpu")))
        model.eval()
        self.model = model
    def __call__(self, batch):
        return [True] * len(batch["id"])
    # Comment this to have an error
    def __reduce__(self):
        return (self.__class__, (self.path_model,))
​
​
dataset = Dataset.from_dict({"id": [0, 1, 2, 4]})
​
# Download (100 MB) at https://github.com/emiliantolo/pytorch_nsfw_model/raw/master/ResNet50_nsfw_model.pth
path_model = "/fsx/hugo/nsfw_image/ResNet50_nsfw_model.pth"
​
filter_function = FilterFunction(path_model=path_model)
​
# Works
filtered_dataset = dataset.filter(filter_function, num_proc=1, batched=True, batch_size=2)
# Doesn't work
filtered_dataset = dataset.filter(filter_function, num_proc=2, batched=True, batch_size=2)
```

### Expected behavior

The command `filtered_dataset = dataset.filter(filter_function, num_proc=2, batched=True, batch_size=2)` should work and not hang.

### Environment info

Datasets: 2.11.0
Pyarrow: 11.0.0
Ubuntu
Unsupported data files raise TypeError: 'NoneType' object is not iterable
Author: albertvillanova | Pull request: no | State: closed

Currently, we raise a TypeError for unsupported data files:
```
TypeError: 'NoneType' object is not iterable
```
See:
- https://github.com/huggingface/datasets-server/issues/1073

We should give a more informative error message.
Raise subprocesses traceback when interrupting
Author: lhoestq | Pull request: yes | State: closed

When a subprocess hangs in `filter` or `map`, one should be able to get the subprocess' traceback when interrupting the main process. Right now it shows nothing.

To do so I `.get()` the subprocesses async results even the main process is stopped with e.g. `KeyboardInterrupt`. I added a timeout in case the subprocess is hanging or crashed.
Offset overflow while doing regex on a text column
Author: nishanthcgit | Pull request: no | State: open

### Describe the bug

`ArrowInvalid: offset overflow while concatenating arrays`

Same error as [here](https://github.com/huggingface/datasets/issues/615)

### Steps to reproduce the bug


Steps to reproduce: (dataset is a few GB big so try in colab maybe)

```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.

### Expected behavior

Dataset should have a new column with processed text

### Environment info

Datasets version - 2.11.0
Error using `load_datasets`
Author: gjyoungjr | Pull request: no | State: closed

### Describe the bug

I tried to load a dataset using the `datasets` library in a conda jupyter notebook and got the below error. 

```
ImportError: dlopen(/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib
  Referenced from: <65B094A2-59D7-31AC-A966-4DB9E11D2A15> /Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so
  Reason: tried: '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file, not in dyld cache)
```

### Steps to reproduce the bug

Run the `load_datasets` function

### Expected behavior

I expected the dataset to be loaded into my notebook.

### Environment info

name: review_sense
channels:
  - apple
  - conda-forge
dependencies:
  - python=3.8
  - pip>=19.0
  - jupyter
  - tensorflow-deps
  #- scikit-learn
  #- scipy
  - pandas
  - pandas-datareader
  - matplotlib
  - pillow
  - tqdm
  - requests
  - h5py
  - pyyaml
  - flask
  - boto3
  - ipykernel
  - seaborn
  - pip:
      - tensorflow-macos==2.9
      - tensorflow-metal==0.5.0
      - bayesian-optimization
      - gym
      - kaggle
      - huggingface_hub
      - datasets
      - numpy
      - huggingface
TypeError: 'NoneType' object does not support item assignment
Author: v-yunbin | Pull request: no | State: closed

command：
```
def load_datasets(formats, data_dir=datadir, data_files=datafile）：
      dataset = load_dataset(formats, data_dir=datadir, data_files=datafile, split=split, streaming=True, **kwargs)
return dataset
raw_datasets = DatasetDict()
raw_datasets["train"] = load_datasets(“csv”,  args.datadir, "train.csv", split=train_split)
raw_datasets["test"] = load_datasets(“csv”, args.datadir, "dev.csv", split=test_split)
raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
```

error：
```
    main()
  File "peft_adalora_whisper_large_training.py", line 502, in main
    raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/datasets/dataset_dict.py", line 2015, in cast_column
    info.features[column] = feature
TypeError: 'NoneType' object does not support item assignment
```
Schrödinger's dataset_dict
Author: liujuncn | Pull request: no | State: closed

### Describe the bug

If you use load_dataset('json', data_files="path/test.json"), it will return DatasetDict({train:...}).

And if you use load_dataset("path"), it will return DatasetDict({test:...}).

Why can't the output behavior be unified?

### Steps to reproduce the bug

as description above.

### Expected behavior

consistent predictable output.

### Environment info

'2.11.0'
datasets.load_dataset("code_search_net", "python") : NotADirectoryError: [Errno 20] Not a directory
Author: jason-brian-anderson | Pull request: no | State: closed

### Describe the bug

While checking out the [tokenizer tutorial](https://huggingface.co/course/chapter6/2?fw=pt), i noticed getting an error while initially downloading the python dataset used in the examples.

The [collab with the error is here](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter6/section2.ipynb#scrollTo=hGb69Yo3eV8S)

```
from datasets import load_dataset
import os

os.environ["HF_DATASETS_CACHE"] = "/workspace"

# This can take a few minutes to load, so grab a coffee or tea while you wait!
raw_datasets = load_dataset("code_search_net", "python")
```

yeilds:

```
ile /opt/conda/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:524, in xlistdir(path, use_auth_token)
    522 main_hop, *rest_hops = _as_str(path).split("::")
    523 if is_local_path(main_hop):
--> 524     return os.listdir(path)
    525 else:
    526     # globbing inside a zip in a private repo requires authentication
    527     if not rest_hops and (main_hop.startswith("http://") or main_hop.startswith("https://")):

NotADirectoryError: [Errno 20] Not a directory: '/workspace/downloads/25ceeb4c25ab737d688bd56ea92bfbb1f199fe572470456cf2d675479f342ac7/python/final/jsonl/train'
```

I was able to reproduce this erro both in the collab and on my own pytorch/pytorch container pulled from the dockerhub official pytorch image, so i think it may be a server side thing. 

### Steps to reproduce the bug

Steps to reproduce the issue:

1. run `raw_datasets = load_dataset("code_search_net", "python")`

### Expected behavior

expect the code to not exception during dataset pull.

### Environment info

i tried both the default HF_DATASETS_CACHE on Collab, and on my local container.  i then pointed to the HF_DATASETS_CACHE to a  large capacity local storage and the problem was consisten across all 3 scenarios.
Use Pandas' `read_json` in the JSON builder
Author: mariosasko | Pull request: no | State: open

Instead of PyArrow's `read_json`, we should use `pd.read_json` in the JSON builder for consistency with the CSV and SQL builders (e.g., to address https://github.com/huggingface/datasets/issues/5725).

In Pandas2.0, to get the same performance, we can set the `engine` to "pyarrow". The issue is that Colab still doesn't install Pandas 2.0 by default, so I think it's best to wait for this to be resolved on their side to avoid downgrading decoding performance in scenarios when Pandas 2.0 is not installed.
ArrowDataset.save_to_disk lost some logic of remote
Author: Zoupers | Pull request: no | State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/e7ce0ac60c7efc10886471932854903a7c19f172/src/datasets/arrow_dataset.py#L1371

Here is the bug point, when I want to save from a `DatasetDict` class and the items of the instance is like `[('train', Dataset({features: ..., num_rows: ...}))]` , there is no guarantee that there exists a directory name `train` under `dataset_dict_path`.

### Steps to reproduce the bug

1. Mock a DatasetDict with items like what I said.
2. using save_to_disk with storage_options, u can use local sftp. code may like below

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset.save_to_disk('sftp:///tmp', storage_options={'host': 'localhost', 'username': 'admin'})
```
I suppose u can reproduce the bug by these steps. 

### Expected behavior

Should create the folder if it does not exists, just like we do locally.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-6.2.10-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.13.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Fix JSON builder when missing keys in first row 
Author: albertvillanova | Pull request: yes | State: closed

Until now, the JSON builder only considered the keys present in the first element of the list:
- Either explicitly: by passing index 0 in `dataset[0].keys()`
- Or implicitly: `pa.Table.from_pylist(dataset)`, where "schema (default None): If not passed, will be inferred from the first row of the mapping values"

This PR fixes the bug by considering the union of the keys present in all the rows.


Fix #5726.
Tiktoken tokenizers are not pickable 
Author: markovalexander | Pull request: no | State: closed

### Describe the bug

Since tiktoken tokenizer is not pickable, it is not possible to use it inside `dataset.map()` with multiprocessing enabled. However, you [made](https://github.com/huggingface/datasets/issues/5536) tiktoken's tokenizers pickable in `datasets==2.10.0` for caching. For some reason, this logic does not work in dataset processing and raises `TypeError: cannot pickle 'builtins.CoreBPE' object`


### Steps to reproduce the bug

```
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
    num_proc=2,
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

starts processing dataset 

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-1021-oracle-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.0
load_dataset("squad") doesn't work in 2.7.1 and 2.10.1
Author: yaseen157 | Pull request: no | State: closed

### Describe the bug

There is an issue that seems to be unique to the "squad" dataset, in which it cannot be loaded using standard methods. This issue is most quickly reproduced from the command line, using the HF examples to verify a dataset is loaded properly.

This is not a problem with "squad_v2" dataset for example.

### Steps to reproduce the bug

cmd line
> $ python -c "from datasets import load_dataset; print(load_dataset('squad', split='train')[0])"

OR

Python IDE
> from datasets import load_dataset
> load_dataset("squad")

### Expected behavior

I expected to either see the output described here from running the very same command in command line ([https://huggingface.co/docs/datasets/installation]), or any output that does not raise Python's TypeError.

There is some funky behaviour in the dataset builder portion of the codebase that means it is trying to import the squad dataset with an incorrect path, or the squad dataset couldn't be downloaded. I'm not really sure what the problem is beyond that. Messing around with caching I did manage to get it to load the dataset once, and then couldn't repeat this.

### Environment info

datasets=2.7.1 **or** 2.10.1, python=3.10.8, Linux 3.10.0-1160.36.2.el7.x86_64 **or** Windows 10-64
How to use Distill-BERT with different datasets?
Author: sauravtii | Pull request: no | State: closed

### Describe the bug

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Steps to reproduce the bug

I recently read [this](https://huggingface.co/docs/transformers/quicktour#train-with-tensorflow:~:text=The%20most%20important%20thing%20to%20remember%20is%20you%20need%20to%20instantiate%20a%20tokenizer%20with%20the%20same%20model%20name%20to%20ensure%20you%E2%80%99re%20using%20the%20same%20tokenization%20rules%20a%20model%20was%20pretrained%20with.) and was wondering how to use distill-BERT (which is pre-trained with imdb dataset) with a different dataset (for eg. [this](https://huggingface.co/datasets/yhavinga/imdb_dutch) dataset)?

### Expected behavior

Distill-BERT should work with different datasets.

### Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 11.0.0