SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: One or several metadata.jsonl were found, but not in the same directory or in a parent directory

Comment:
Also, when generated from a zip archive, the dataset contains only a few images. In my case, 20 versus 2000+ contained in the archive. The generation from folders works as expected.
Issue: One or several metadata.jsonl were found, but not in the same directory or in a parent directory

Comment:
Thanks for reporting, @blghtr.

You should include the `metadata.jsonl` in your ZIP archives, at the root level directory.

I agree that our documentation is not clear enough. Maybe we could improve it.
Issue: One or several metadata.jsonl were found, but not in the same directory or in a parent directory

Comment:
You can find a dummy dataset example here: https://huggingface.co/datasets/albertvillanova/tmp-imagefolder-metadata

```
tmp-imagefolder-metadata/
└── data/
    ├── train.zip
    └── valid.zip
```
where, the directory structure within the `train.zip` archive is:
```
metadata.jsonl
train/
    ├── bharatanatyam/
        └── bharatanatyam_original_113.jpg_70c297a2-e2f2-4ed8-b93c-0c03d0809fe2.jpg
    └── kathak/
        └── kathak_original_10.jpg_2c4a2c3d-47fc-4b33-9c09-38b542826632.jpg
```
and the metadata file contains:
```
{"file_name": "train/bharatanatyam/bharatanatyam_original_113.jpg_70c297a2-e2f2-4ed8-b93c-0c03d0809fe2.jpg", "text": "first"}
{"file_name": "train/kathak/kathak_original_10.jpg_2c4a2c3d-47fc-4b33-9c09-38b542826632.jpg", "text": "second"}
```
Issue: Multi-image loading in Imagefolder dataset

Comment:
Supporting this could be useful (I remember a use-case for this on the Hub). Do you agree @polinaeterna? 

Implementing this should be possible if we iterate over metadata files and build image/audio file paths instead of iterating over image/audio files and looking for the corresponding entries in metadata files.
Issue: Multi-image loading in Imagefolder dataset

Comment:
I've build a similar feature from scratch and would be interested to combine it with the datasets package.

My solution works something like this:
Interpret the first element of each column as a file path. If the path exists and is a file, (try to) load the files for the entire column. Thereby, one isn't restricted to a particular column name, with comes in handy when dealing with multiple file columns.

I've looked into the code to try to implement this, but didn't find the right places. I'm also open to contribute, but will need some guidance.
Issue: Can I load in list of list of dict format? 

Comment:
Thanks for reporting, @LZY-the-boys.

Could you please give more details about what is your intended dataset structure? What are the names of the columns and the value of each row?

Currently, the JSON-Lines format is supported:
- Each line correspond to one row of the dataset
- Each line is composed of one JSON object, where the names are the names of the columns, and the values are the values for the row-column pair.
Issue: Calling shuffle on a IterableDataset with streaming=True, gives "ValueError: cannot reshape array"

Comment:
Thanks, this appears to have fixed the issue.

I've created a PR for the same change in the mnist dataset: https://huggingface.co/datasets/mnist/discussions/3/files
Issue: [IterableDatasets] Add column followed by interleave datasets gives bogus outputs

Comment:
Problem with the code snippet! Using global vars and functions was not a good idea with iterable datasets!

If we update to:
```python
from datasets import load_dataset

original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# now add a new column to our streaming dataset using our hack
name = "new_column"
column_1 = [f"new dataset 1, row {i}" for i in range(50)]

new_features = original_dataset.features.copy()
new_features[name] = new_features["file"]  # I know that "file" has the right column type to match our new feature

def add_column_fn_1(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column_1[idx]}

modified_dataset_1 = original_dataset.map(add_column_fn_1, with_indices=True, features=new_features)

# now create a second modified dataset using the same trick
column_2 = [f"new dataset 2, row {i}" for i in range(50)]

def add_column_fn_2(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column_2[idx]}

modified_dataset_2 = original_dataset.map(add_column_fn_2, with_indices=True, features=new_features)

interleaved_dataset = interleave_datasets([modified_dataset_1, modified_dataset_2])

for i, sample in enumerate(interleaved_dataset):
    print(sample["new_column"])
    if i == 10:
        break
```
we get the correct outputs:
```python
new dataset 1, row 0
new dataset 2, row 0
new dataset 1, row 1
new dataset 2, row 1
new dataset 1, row 2
new dataset 2, row 2
new dataset 1, row 3
new dataset 2, row 3
new dataset 1, row 4
new dataset 2, row 4
new dataset 1, row 5
```
Issue: Streaming dataset looses `.feature` method after `.add_column`

Comment:
I believe the issue resides in this line:
https://github.com/huggingface/datasets/blob/7c3a9b057c476c40d157bd7a5d57f49066239df0/src/datasets/iterable_dataset.py#L1415

If we pass the **new** features of the dataset to the `.map` method we can return the features after adding a column, e.g.:
```python
from datasets import load_dataset, Value

original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
print(original_dataset.features.keys())

# now add a new column to our streaming dataset using our hack
name = "new_column"
column = ["some random text" for _ in range(50)]

new_features = original_dataset.features.copy()
new_features[name] = Value(dtype="string", id=None)  # I know the correct column type for this feature

def add_column_fn(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column[idx]}

modified_dataset = original_dataset.map(add_column_fn, with_indices=True, features=new_features)

print(modified_dataset.features.keys())
```
**Print Output:**
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id', 'new_column'])
```
Issue: Fail to create datasets from a generator when using Google Big Query

Comment:
`from_generator` expects a generator function, not a generator object, so this should work:
```python
from datasets import Dataset
from google.cloud import bigquery

client = bigquery.Client()

def gen()
    # Perform a query.
    QUERY = (
        'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
         'WHERE state = "TX" '
         'LIMIT 100')
     query_job = client.query(QUERY)  # API request
     yield from query_job.result()  # Waits for query to finish

ds = Dataset.from_generator(rows)

for r in ds:
    print(r)
```
Issue: Fail to create datasets from a generator when using Google Big Query

Comment:
@mariosasko your code was incomplete, so I tried to fix it:

```py
from datasets import Dataset
from google.cloud import bigquery

client = bigquery.Client()

def gen():
    # Perform a query.
    QUERY = (
        'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
         'WHERE state = "TX" '
         'LIMIT 100')
    query_job = client.query(QUERY)  # API request
    yield from query_job.result()  # Waits for query to finish

ds = Dataset.from_generator(gen)

for r in ds:
    print(r)
```

The error is also present in this case:

```
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
```

I think it doesn't matter if the generator is an object or a function. The problem is that the generator is referencing an object that is not pickable (the client in this case).
Issue: Fail to create datasets from a generator when using Google Big Query

Comment:
It does matter: this function expects a generator function, as stated in the docs.

This should work:
```python
from datasets import Dataset
from google.cloud import bigquery

def gen():
    client = bigquery.Client()
    # Perform a query.
    QUERY = (
        'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
         'WHERE state = "TX" '
         'LIMIT 100')
    query_job = client.query(QUERY)  # API request
    yield from query_job.result()  # Waits for query to finish

ds = Dataset.from_generator(gen)

for r in ds:
    print(r)
```

We could allow passing non-picklable objects and use a random hash for the generated arrow file. In that case, the caching mechanism would not work, meaning repeated calls with the same set of arguments would generate new datasets instead of reusing the cached version, but this behavior is still better than raising an error.
Issue: Fail to create datasets from a generator when using Google Big Query

Comment:
Thank you @mariosasko . Your last code is working indeed. Curiously, the important detail here was to wrap the client instantiation within the generator itself. If the line `client = bigquery.Client()` is moved outside, then the error is back.

I see now also your point in regard to the generator being a generator function. We can close the issue if you want.
Issue: AttributeError: 'Version' object has no attribute 'match'

Comment:
I got the same error, and the official website for visual genome is down. Did you solve this problem?
Issue: AttributeError: 'Version' object has no attribute 'match'

Comment:
Apart form data host server being down, there is an additional issue with the `datasets` library introduced by this PR:
- #5238

I am working to fix it.
Issue: AttributeError: 'Version' object has no attribute 'match'

Comment:
For the issue with their data host server being down, I have opened a discussion in the "Community" tab of the Hub dataset: https://huggingface.co/datasets/visual_genome/discussions/3
Let's continue the discussion there.
Issue: AttributeError: 'Version' object has no attribute 'match'

Comment:
The authors just replied to us with their new URL: https://homes.cs.washington.edu/~ranjay/visualgenome/

We have fixed the datasets loading script, which is operative again.
Issue: [BUG] With Pandas 2.0.0, `load_dataset` raises `TypeError: read_csv() got an unexpected keyword argument 'mangle_dupe_cols'`

Comment:
Thanks for reporting, @keyboardAnt.

We haven't noticed any crash in our CI tests. Could you please indicate specifically the `load_dataset` command that crashes in your side, so that we can reproduce it?
Issue: dataclass.py in virtual environment is   overriding the stdlib module "dataclasses"

Comment:
We no longer depend on `dataclasses` (for almost a year), so I don't think our package is the problematic one. 

I think it makes more sense to raise this issue in the `dataclasses` repo: https://github.com/ericvsmith/dataclasses.
Issue: weird result during dataset split when data path starts with `/data`

Comment:
hi! 
I think you can run python from `/data/train/raw/` directory and load dataset as `load_dataset("code_contests")` to mitigate this issue as a workaround. 
@ericxsun Do you want to open a PR to fix the regex? As you already found the solution :)
Issue: weird result during dataset split when data path starts with `/data`

Comment:
> hi! I think you can run python from `/data/train/raw/` directory and load dataset as `load_dataset("code_contests")` to mitigate this issue as a workaround. @ericxsun Do you want to open a PR to fix the regex? As you already found the solution :)

Sure, please see https://github.com/huggingface/datasets/pull/5748  @polinaeterna
Issue: weird result during dataset split when data path starts with `/data`

Comment:
I think `string_to_dict` is ok, and that the issue is that it gets `'/data2/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'` as input instead of `'data/test-00000-of-00001-9c49eeff30aacaa8.parquet'`. The path should be relative to the directory being loaded by `load_dataset`
Issue: load_dataset("text","dataset.txt") loads the wrong dataset!

Comment:
You need to provide a text file as `data_files`, not as a configuration:

```python
my_dataset = load_dataset("text", data_files="TextFile.txt")
```

Otherwise, since `data_files` is `None`, it picks up Colab's sample datasets from the `content` dir.
Issue: ClassLabel Error 

Comment:
Hi, you can use the `cast_column` function to change the feature type from a `Value(int64)` to `ClassLabel`:

```py
dataset = dataset.cast_column("label", ClassLabel(names=["label_1", "label_2", "label_3"]))
print(dataset.features)
{'text': Value(dtype='string', id=None),
 'label': ClassLabel(names=['label_1', 'label_2', 'label_3'], id=None)}
```
Issue: FORCE_REDOWNLOAD raises "Directory not empty" exception on second run

Comment:
Hi ! I couldn't reproduce your issue :/

It seems that `shutil.rmtree` failed. It is supposed to work even if the directory is not empty, but you still end up with `OSError: [Errno 39] Directory not empty:`. Can you make sure another process is not using this directory at the same time ?
Issue: Enwik8 should support the standard split

Comment:
The Enwik8 pipeline is not present in this codebase, and is hosted elsewhere. I have opened a PR [there](https://huggingface.co/datasets/enwik8/discussions/4) instead.
Issue: load_dataset fails with FileNotFound error on Windows

Comment:
Hi! Can you please paste the entire error stack trace, not only the last few lines?
Issue: load_dataset fails with FileNotFound error on Windows

Comment:
`----> 1 dataset = datasets.load_dataset("glue", "ax")

File ~\anaconda3\envs\huggingface\Lib\site-packages\datasets\load.py:1767, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   1762 verification_mode = VerificationMode(
   1763     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   1764 )
   1766 # Create a dataset builder
-> 1767 builder_instance = load_dataset_builder(
   1768     path=path,
   1769     name=name,
   1770     data_dir=data_dir,
   1771     data_files=data_files,
   1772     cache_dir=cache_dir,
   1773     features=features,
   1774     download_config=download_config,
   1775     download_mode=download_mode,
   1776     revision=revision,
   1777     use_auth_token=use_auth_token,
   1778     storage_options=storage_options,
   1779     **config_kwargs,
   1780 )
   1782 # Return iterable dataset in case of streaming
   1783 if streaming:

File ~\anaconda3\envs\huggingface\Lib\site-packages\datasets\load.py:1498, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, storage_options, **config_kwargs)
   1496     download_config = download_config.copy() if download_config else DownloadConfig()
   1497     download_config.use_auth_token = use_auth_token
-> 1498 dataset_module = dataset_module_factory(
   1499     path,
   1500     revision=revision,
   1501     download_config=download_config,
   1502     download_mode=download_mode,
   1503     data_dir=data_dir,
   1504     data_files=data_files,
   1505 )
   1507 # Get dataset builder class from the processing script
   1508 builder_cls = import_main_class(dataset_module.module_path)

File ~\anaconda3\envs\huggingface\Lib\site-packages\datasets\load.py:1211, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1209                 raise e1 from None
   1210             if isinstance(e1, FileNotFoundError):
-> 1211                 raise FileNotFoundError(
   1212                     f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1213                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1214                 ) from None
   1215             raise e1 from None
   1216 else:`
Issue: load_dataset fails with FileNotFound error on Windows

Comment:
Okay, this is the issue:
```
FileNotFoundError: [WinError 3] The system cannot find the path specified: 
'C:\\Users\\...\\.cache\\huggingface'
``` 

I don't remember seeing this error before.

I guess it could happen in a multi-process environment if one of the processes deletes the `datasets` cache as the other one is loading a dataset (with `load_dataset`), so make sure that's not the case. Also, you can disable the Windows max path length limit (if enabled), but this is most likely not the problem.
Issue: How to limit the number of examples in dataset, for testing?

Comment:
Hi! You can use the `nrows` parameter for this:
```python
data = load_dataset("json", data_files=data_path, nrows=10)
```
Issue: How to limit the number of examples in dataset, for testing?

Comment:
I misread the format in which the dataset is stored -  the `nrows` parameter works for CSV, but not JSON.

This means the only option is first to create a DataFrame and then convert it to a Dataset object:
```python
import pandas as pd
from datasets import Dataset

df = pd.read_json(data_path, lines=True, nrows=10)
ds = Dataset.from_pandas(df)
```
Issue: Error after shuffling streaming IterableDatasets with downloaded dataset

Comment:
Moving `"en"` to the end of the path instead of passing it as a config name should fix the error:
```python
import datasets
dataset = datasets.load_dataset('/path/to/your/data/dir/en', streaming=True, split='train')
dataset = dataset.shuffle(buffer_size=10_000, seed=42)
next(iter(dataset))
```

PS: https://github.com/huggingface/datasets/pull/5331, once merged, will allow us to define C4's configs in its README, making downloading it much more user-friendly.
Issue: Distributed Training Error on Customized Dataset

Comment:
Hmm the error doesn't seem related to data loading.

Regarding `split_dataset_by_node`: it's generally used to split an iterable dataset (e.g. when streaming) in pytorch DDP. It's not needed if you use a regular dataset since the pytorch DataLoader already assigns a subset of the dataset indices to each node.
Issue: Streaming IterableDatasets do not work with torch DataLoaders

Comment:
I'm experiencing the same problem that @jlehrer1. I was able to reproduce it with a very small example:

```py
from datasets import Dataset, load_dataset, load_dataset_builder
from torch.utils.data import DataLoader


def my_gen():
    for i in range(1, 4):
        yield {"a": i}

# Saving the dataset as a parquet file
dataset = Dataset.from_generator(my_gen)
train_path = "/tmp/test.parquet"
dataset.to_parquet(train_path)

# Creating a local dataset from the parquet file
data_files = {"train": [str(train_path)]}
builder = load_dataset_builder("parquet", data_files=data_files)
builder.download_and_prepare("/tmp/test_ds", file_format="parquet")

# Loading the dataset from the local directory as streaming
dataset = load_dataset("parquet", data_dir="/tmp/test_ds", split="train", streaming=True)
dataset.with_format("torch")

dl = DataLoader(dataset, batch_size=2, num_workers=1)
for row in dl:
    print(row)
```

My env info:
```
datasets                      2.11.0
torch                         2.0.0
torchvision                   0.15.1
Python 3.9.16
```

Note that the example above doesn't fail if the number of workers used is `0`
Issue: Streaming IterableDatasets do not work with torch DataLoaders

Comment:
I cannot reproduce this error, not even with your MRE @ivanprado (your env appears to be the same as Colab's, and your code runs there without issues).
Issue: Streaming IterableDatasets do not work with torch DataLoaders

Comment:
@mariosasko you are right, it works on Colab. I digged deeper and found that the problem arises when the multiprocessing method is set to be `spawn`. This code reproduces the problem in Colab:

```py
from datasets import Dataset, load_dataset, load_dataset_builder
from torch.utils.data import DataLoader
import multiprocessing as mp

mp.set_start_method('spawn')

def my_gen():
    for i in range(1, 4):
        yield {"a": i}


def main():
    # Saving the dataset as a parquet file
    dataset = Dataset.from_generator(my_gen)
    train_path = "/tmp/test.parquet"
    dataset.to_parquet(train_path)

    # Creating a local dataset from the parquet file
    data_files = {"train": [str(train_path)]}
    builder = load_dataset_builder("parquet", data_files=data_files)
    builder.download_and_prepare("/tmp/test_ds", file_format="parquet")

    # Loading the dataset from the local directory as streaming
    dataset = load_dataset("parquet", data_dir="/tmp/test_ds", split="train", streaming=True)
    dataset.with_format("torch")

    dl = DataLoader(dataset, batch_size=2, num_workers=1)
    for row in dl:
        print(row)

main()
```
Issue: Streaming IterableDatasets do not work with torch DataLoaders

Comment:
So is there a way to fix this by changing the `mp` method? This is blocking any usage of the `datasets` library for me
Issue: Streaming IterableDatasets do not work with torch DataLoaders

Comment:
@jlehrer1 can you try adding `mp.set_start_method('fork')` at the beginning of your code? Maybe this helps you. Keep us posted.
Issue: Array2D feature creates a list of list instead of a numpy array

Comment:
Hi! 

You need to set the format to `np` before indexing the dataset to get NumPy arrays:
```python
features = Features(dict(seq=Array2D((2,2), 'float32')))  
ds = Dataset.from_dict(dict(seq=[np.random.rand(2,2)]), features=features)
ds.set_format("np")
a = ds[0]['seq']
```

>  I think it should not be the expected behavior especially when I feed a numpy array as input to the data creation function. Why is it converting my array into a list?

The same dataset can have examples in different types (Numpy arrays, Torch tensors, Pandas series, etc.), so recovering them all would be slow and impractical. Instead, the design of our formatting API is similar to Arrow's (the lib we use internally to store data on disk/ in RAM), which allows converting a batch of data to Python/Numpy/Pandas in a single call (and uses C++ to do so to make it faster).

> Also if I change the first dimension of the Array2D shape to None, it's returning array correctly.

Setting the first dimension to `None` makes it variable-length (allows passing arrays with the first dimensions of differing lengths).
Issue: Array2D feature creates a list of list instead of a numpy array

Comment:
Current behavior when indexing the dataset:
- Using `Array((2,2))` returns a list of lists.
- Using `Array((None,2))` returns a numpy array.

Don't you think this is kind of unexpected behavior from end-user perspective? 
As a user, I expect that when I use `Array2D`, the behavior needs to be consistent even if I specify None or not. It should either return a list or an array. It needs to choose one. Let's say if it always return a list, then I will call `ds.set_format('np')` no problem.

The consistency can be in any of these aspects:
1. preserves the type of the input data (in this case, a numpy array)
2. ensure the output type is always the same (it can be either list or array, but it needs to be one of them)

Right now the API doesn't conform to any of these aspects. But I think it needs to conform to one.