liuqingwen/github-issues huggingface dataset:
load_dataset: TypeError: 'NoneType' object is not callable, on local dataset filename changes

### Describe the bug

I've adapted Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py) to train using a local dataset, which has been working. Upon changing the filenames of the `.json` & `.py` files in my local dataset directory, `dataset = load_dataset(path_or_dataset)["train"]` throws the error:

```python
2023-04-30 09:10:52 INFO [training.trainer] Loading dataset from dushowxa-characters
Traceback (most recent call last):
  File "/data/dushowxa-dolly/train_dushowxa.py", line 26, in <module>
    load_training_dataset()
  File "/data/dushowxa-dolly/training/trainer.py", line 89, in load_training_dataset
    dataset = load_dataset(path_or_dataset)["train"]
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1528, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
```
The local dataset filenames were of the form `dushowxa-characters/expanse-dushowxa-characters.json` and are now of the form `dushowxa-characters/dushowxa-characters.json` (the word `expanse-` was removed from the filenames). Is this perhaps a dataset caching issue?

I have attempted to manually clear caches, but to no effect:

```sh
rm -rfv ~/.cache/huggingface/datasets/*
rm -rfv ~/.cache/huggingface/modules/* 
```

### Steps to reproduce the bug

Run `python3 train_dushowxa.py` (adapted from Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py)).

### Expected behavior

Training succeeds as before local dataset filenames were changed.

### Environment info

Ubuntu 22.04, Python 3.10.6, venv
```python
accelerate>=0.16.0,<1
click>=8.0.4,<9
datasets>=2.10.0,<3
deepspeed>=0.9.0,<1
transformers[torch]>=4.28.1,<5
langchain>=0.0.139
```

Comments:
This error means a `DatasetBuilder` subclass that generates the dataset could not be found inside the script, so make sure `dushowxa-characters/dushowxa-characters.py `is a valid dataset script (assuming `path_or_dataset` is `dushowxa-characters`)

Also, we should improve the error to make it more obvious what the problem is.
Return the name of the currently loaded file in the load_dataset function.
Labels: name: enhancement
description: New feature or request

name: good first issue
description: Good for newcomers

### Feature request

Add an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

### Motivation

When training large language models, machine problems may interrupt the training process. In such cases, it is common to load a previously saved checkpoint to resume training. I would like to be able to obtain the names of the previously trained data shards, so that I can skip these parts of the data during continued training to avoid overfitting and redundant training time.

### Your contribution

I currently use a dataset in jsonl format, so I am primarily interested in the json format. I suggest adding the file name to the returned table here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.

Comments:
Implementing this makes sense (e.g., `tensorflow_datasets`' imagefolder returns image filenames). Also, in Datasets 3.0, we plan only to store the bytes of an image/audio, not its path, so this feature would be useful when the path info is still needed.

Hey @mariosasko, Can I work on this issue, this one seems interesting to implement. I have contributed to jupyterlab recently, and would love to contribute here as well. 

@tsabbir96 if you are planning to start working on this, you can take on this issue by writing a comment with only the keyword: #self-assign

#self-assign

@albertvillanova thank you for letting me contribute here. 
@albertvillanova @mariosasko As I am totally new to this repo, could you tell me something more about this issue or perhaps give me some idea on how I can proceed with it? Thanks!
Improve `Create a dataset` tutorial 
Labels: name: documentation
description: Improvements or additions to documentation

Our [tutorial on how to create a dataset](https://huggingface.co/docs/datasets/create_dataset) is a bit misleading. 
1. In **Folder-based builders** section it says that we have two folder-based builders as standard builders, but we also have similar builders (that can be created from directory with data of required format) for `csv`, `json/jsonl`, `parquet` and `txt` files. We have info about these loaders in separate [guide for loading](https://huggingface.co/docs/datasets/loading#local-and-remote-files) but it's worth briefly mentioning them in the beginning tutorial because they are more common and for consistency. Would be helpful to add the link to the full guide.
2. **From local files** section lists methods for creating a dataset from in-memory data which are also described in [loading guide](https://huggingface.co/docs/datasets/loading#inmemory-data).  

Maybe we should actually rethink and restructure this tutorial somehow.

Comments:
I can work on this. The link to the tutorial seems to be broken though @polinaeterna. 

@isunitha98selvan would be great, thank you! which link are you talking about? I think it should work: https://huggingface.co/docs/datasets/create_dataset
Files downloaded to cache do not respect umask
Labels: name: bug
description: Something isn't working

As reported by @stas00, files downloaded to the cache do not respect umask:
```bash
$ ls -l /path/to/cache/datasets/downloads/
-rw------- 1 uername    username    150M Apr 25 16:41 5e646c1d600f065adaeb134e536f6f2f296a6d804bd1f0e1fdcd20ee28c185c6 
```

Related to:
- #2065

Comments:
CI ZeroDivisionError
Labels: name: bug
description: Something isn't working

Sometimes when running our CI on Windows, we get a ZeroDivisionError:
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - ZeroDivisionError: float division by zero
```

See for example:
- https://github.com/huggingface/datasets/actions/runs/4809358266/jobs/8560513110
- https://github.com/huggingface/datasets/actions/runs/4798359836/jobs/8536573688

```
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

split = 'test', start_time = 1682516718.8236516, num_samples = 2, num_steps = 1

    def speed_metrics(split, start_time, num_samples=None, num_steps=None):
        """
        Measure and return speed performance metrics.
    
        This function requires a time snapshot `start_time` before the operation to be measured starts and this function
        should be run immediately after the operation to be measured has completed.
    
        Args:
        - split: name to prefix metric (like train, eval, test...)
        - start_time: operation start time
        - num_samples: number of samples processed
        """
        runtime = time.time() - start_time
        result = {f"{split}_runtime": round(runtime, 4)}
        if num_samples is not None:
>           samples_per_second = num_samples / runtime
E           ZeroDivisionError: float division by zero

C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\transformers\trainer_utils.py:354: ZeroDivisionError
```

Comments:
IterableDataset.with_format("torch") not working
Labels: name: bug
description: Something isn't working

name: enhancement
description: New feature or request

name: streaming

### Describe the bug

After calling the with_format("torch") method on an IterableDataset instance, the data format is unchanged.

### Steps to reproduce the bug

```python
from datasets import IterableDataset
def gen():
    for i in range(4):
        yield {"a": [i] * 4}
dataset = IterableDataset.from_generator(gen).with_format("torch")
next(iter(dataset))
```

### Expected behavior

`{"a": torch.tensor([0, 0, 0, 0])}` is expected, but `{"a": [0, 0, 0, 0]}` is observed.

### Environment info

```bash
platform==ubuntu 22.04.01
python==3.10.9
datasets==2.11.0
```

Comments:
Hi ! Thanks for reporting, I'm working on it ;)
TIFF/TIF support 
Labels: name: enhancement
description: New feature or request

### Feature request

I currently have a dataset (with tiff and json files) where I have to do this:

`wget path_to_data/images.zip && unzip images.zip`

`wget path_to_data/annotations.zip && unzip annotations.zip`
 
 Would it make sense a contribution that supports these type of files? 

### Motivation

instead of using `load_dataset` have to use wget as these files are not supported for annotations with JSON and images with TIFF files. 


Additionally to this, the PIL formatting from datasets does not read correctly the image channels with TIFF format, besides multichannel adaptation might be necessary as well (as my data e.g has more than 3 channels)

### Your contribution

1. Support TIFF images over multi channel format
2. Support JSON annotations

Comments:
The issue with multichannel TIFF images has already been reported in Pillow (https://github.com/python-pillow/Pillow/issues/1888). We can't do much about it on our side.

Still, to avoid the error, you can bypass the default Pillow decoding and define a custom one as follows:
```python
import tifffile # pip install tifffile

dset = dset.cast_column("image", datasets.Image(decode=False))

def decode_mutlichannel_tiff(batch):
    batch["image"] = [tifffile.imread(image["path"]) for image in batch["image"]]
    return batch

dset.set_transform(decode_mutlichannel_tiff)
```

Regarding the annotations, in which format are they? In the COCO format? I think this is a bit too specific to have a built-in loader for it.
Support streaming datasets that use jsonlines
Labels: name: enhancement
description: New feature or request

Extend support for streaming datasets that use `jsonlines.open`.

Currently, if `jsonlines` is installed, `datasets` raises a `FileNotFoundError`:
```
FileNotFoundError: [Errno 2] No such file or directory: 'https://...'
```

See:
- https://huggingface.co/datasets/masakhane/afriqa/discussions/1

Comments:
Unsupported data files raise TypeError: 'NoneType' object is not iterable
Labels: name: bug
description: Something isn't working

Currently, we raise a TypeError for unsupported data files:
```
TypeError: 'NoneType' object is not iterable
```
See:
- https://github.com/huggingface/datasets-server/issues/1073

We should give a more informative error message.

Comments:
Offset overflow while doing regex on a text column

### Describe the bug

`ArrowInvalid: offset overflow while concatenating arrays`

Same error as [here](https://github.com/huggingface/datasets/issues/615)

### Steps to reproduce the bug


Steps to reproduce: (dataset is a few GB big so try in colab maybe)

```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.

### Expected behavior

Dataset should have a new column with processed text

### Environment info

Datasets version - 2.11.0

Comments:
Hi! This looks like an Arrow bug, but it can be avoided by reducing the `writer_batch_size`.

(`ds = ds.map(get_text_caption, writer_batch_size=100)` in Colab runs without issues)
Use Pandas' `read_json` in the JSON builder
Labels: name: enhancement
description: New feature or request

Instead of PyArrow's `read_json`, we should use `pd.read_json` in the JSON builder for consistency with the CSV and SQL builders (e.g., to address https://github.com/huggingface/datasets/issues/5725).

In Pandas2.0, to get the same performance, we can set the `engine` to "pyarrow". The issue is that Colab still doesn't install Pandas 2.0 by default, so I think it's best to wait for this to be resolved on their side to avoid downgrading decoding performance in scenarios when Pandas 2.0 is not installed.

Comments:
train_dataset does not implement __len__

when train using data  precessored by the datasets, I get follow warning and it leads to that I can not set epoch numbers:
`ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.`

Comments:
Thanks for reporting, @v-yunbin.

Could you please give more details, the steps to reproduce the bug, the complete error back trace and the environment information (`datasets-cli env`)?

this is a detail error info from transformers：
```
Traceback (most recent call last):
  File "finetune.py", line 177, in <module>
    whisper_finetune(traindir,devdir,outdir)
  File "finetune.py", line 161, in whisper_finetune
    trainer = Seq2SeqTrainer(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 56, in __init__
    super().__init__(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer.py", line 567, in __init__
    raise ValueError(
ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.
```


How did you create `train_dataset`? The `datasets` library does not appear in your stack trace.

We need more information in order to reproduce the issue...

```
def asr_dataset(traindir,devdir):
    we_voice = IterableDatasetDict()
    #we_voice["train"] = load_from_disk(traindir,streaming=True)
    #we_voice["test"]= load_from_disk(devdir,streaming=True)
    we_voice["train"] = load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train",streaming=True)
    #print(load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train"))
    we_voice["test"] = load_dataset("csv",data_files=os.path.join(devdir,"dev.csv"), split="train",streaming=True)
    we_voice = we_voice.remove_columns(["id"])
    we_voice = we_voice.cast_column("audio", Audio(sampling_rate=16000))
    return we_voice

```

As you are using iterable datasets (`streaming=True`), their length is not defined.

You should:
- Either use non-iterable datasets, which have a defined length: use `DatasetDict` and not passing `streaming=True`
- Or pass `args.max_steps` to the `Trainer`

I don't know how to give a reasonable args.max_steps...........................

Then you should not use streaming.

@albertvillanova I think @v-yunbin, myself, and others might be slightly confused about max_steps and how it differs from num_train_epochs.
Support cloud storage for loading datasets
Labels: name: duplicate
description: This issue or pull request already exists

name: enhancement
description: New feature or request

### Feature request

It seems that the the current implementation supports cloud storage only for `load_from_disk`. It would be nice if a similar functionality existed in `load_dataset`.

### Motivation

Motivation is pretty clear -- let users work with datasets located in the cloud. 

### Your contribution

I can help implementing this. 

Comments:
A duplicate of https://github.com/huggingface/datasets/issues/5281
Multi-image loading in Imagefolder dataset
Labels: name: enhancement
description: New feature or request

### Feature request

Extend the `imagefolder` dataloading script to support loading multiple images per dataset entry.

This only really makes sense if a metadata file is present.

Currently you can use the following format (example `metadata.jsonl`:
```
{'file_name': 'path_to_image.png', 'metadata': ...}
...
```
which will return a batch with key `image` and any other metadata.

I would propose extending `file_name` to also accept a list of files, which would return a batch with key `images` and any other metadata.

### Motivation

This is useful for example in segmentation tasks in computer vision models, or in text-to-image models that also accept conditioning signals such as another image, feature map, or similar. Currently if I want to do this, I would need to write a custom dataset, rather than just use `imagefolder`.

### Your contribution

Would be open to doing a PR, but also happy for someone else to take it as I am not familiar with the datasets library.

Comments:
Supporting this could be useful (I remember a use-case for this on the Hub). Do you agree @polinaeterna? 

Implementing this should be possible if we iterate over metadata files and build image/audio file paths instead of iterating over image/audio files and looking for the corresponding entries in metadata files.

I've build a similar feature from scratch and would be interested to combine it with the datasets package.

My solution works something like this:
Interpret the first element of each column as a file path. If the path exists and is a file, (try to) load the files for the entire column. Thereby, one isn't restricted to a particular column name, with comes in handy when dealing with multiple file columns.

I've looked into the code to try to implement this, but didn't find the right places. I'm also open to contribute, but will need some guidance.
Can I load in list of list of dict format? 
Labels: name: enhancement
description: New feature or request

### Feature request

my jsonl dataset has following format:
```
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
```

I try to use `datasets.load_dataset('json', data_files=path)` or `datasets.Dataset.from_json`, it raises
```
  File "site-packages/datasets/arrow_dataset.py", line 1078, in from_json
    ).read()
  File "site-packages/datasets/io/json.py", line 59, in read
    self.builder.download_and_prepare(
  File "site-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "site-packages/datasets/builder.py", line 967, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "site-packages/datasets/builder.py", line 1749, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "site-packages/datasets/builder.py", line 1892, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Motivation

I wanna use features like `Datasets.map` or `Datasets.shuffle`, so i need the dataset in memory to be `arrow_dataset.Datasets` format

### Your contribution

PR

Comments:
Thanks for reporting, @LZY-the-boys.

Could you please give more details about what is your intended dataset structure? What are the names of the columns and the value of each row?

Currently, the JSON-Lines format is supported:
- Each line correspond to one row of the dataset
- Each line is composed of one JSON object, where the names are the names of the columns, and the values are the values for the row-column pair.
Streaming dataset looses `.feature` method after `.add_column`
Labels: name: bug
description: Something isn't working

### Describe the bug

After appending a new column to a streaming dataset using `.add_column`, we can no longer access the list of dataset features using the `.feature` method.

### Steps to reproduce the bug

```python
from datasets import load_dataset

original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
print(original_dataset.features.keys())

# now add a new column to our streaming dataset
modified_dataset = original_dataset.add_column("new_column", ["some random text" for _ in range(50)])
print(modified_dataset.features.keys())
```
**Print Output:**
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 8
      6 # now add a new column to our streaming dataset
      7 modified_dataset = original_dataset.add_column("new_column", ["some random text" for _ in range(50)])
----> 8 print(modified_dataset.features.keys())

AttributeError: 'NoneType' object has no attribute 'keys'
```

We see that we get the features for the original dataset, but not the modified one with the added column.


### Expected behavior

Features should be persevered after adding a new column, i.e. calling:
```python
print(modified_dataset.features.keys())
```

Should return:
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id', 'new_column'])
```

### Environment info

- `datasets` version: 2.10.2.dev0
- Platform: Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.13.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

Comments:
I believe the issue resides in this line:
https://github.com/huggingface/datasets/blob/7c3a9b057c476c40d157bd7a5d57f49066239df0/src/datasets/iterable_dataset.py#L1415

If we pass the **new** features of the dataset to the `.map` method we can return the features after adding a column, e.g.:
```python
from datasets import load_dataset, Value

original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
print(original_dataset.features.keys())

# now add a new column to our streaming dataset using our hack
name = "new_column"
column = ["some random text" for _ in range(50)]

new_features = original_dataset.features.copy()
new_features[name] = Value(dtype="string", id=None)  # I know the correct column type for this feature

def add_column_fn(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column[idx]}

modified_dataset = original_dataset.map(add_column_fn, with_indices=True, features=new_features)

print(modified_dataset.features.keys())
```
**Print Output:**
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id', 'new_column'])
```
[BUG] With Pandas 2.0.0, `load_dataset` raises `TypeError: read_csv() got an unexpected keyword argument 'mangle_dupe_cols'`

The `load_dataset` function with Pandas `1.5.3` has no issue (just a FutureWarning) but crashes with Pandas `2.0.0`.
For your convenience, I opened a draft Pull Request to fix it quickly: https://github.com/huggingface/datasets/pull/5745

---

* The FutureWarning mentioned above: 
```
FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
```

Comments:
Thanks for reporting, @keyboardAnt.

We haven't noticed any crash in our CI tests. Could you please indicate specifically the `load_dataset` command that crashes in your side, so that we can reproduce it?

This has been fixed in `datasets` 2.11