SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Comment:
Hi @frgfm, thanks for reporting.

As the error message says, streaming mode is not supported out of the box when the dataset contains TAR archive files.

To make the dataset streamable, you have to use `dl_manager.iter_archive`.

There are several examples in other datasets, e.g. food101: https://huggingface.co/datasets/food101/blob/main/food101.py

And yes, as the link you pointed out, for the streaming to be possible, the metadata file must be loaded before all of the images:
- either this is the case when iterating the archive (and you get the metadata file before the images)
- or you have to extract the metadata file by hand and upload it separately to the Hub
Issue: Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Comment:
Hi @albertvillanova :wave:

Thanks! Yeah I saw that but since I didn't have any metadata, I wasn't sure whether I should create them myself.

So one last question:
What is the metadata supposed to be for archives? The relative path of all files in it?
_(Sorry I'm a bit confused since it's quite hard to debug using the single error message from the data preview :sweat_smile: )_
Issue: Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Comment:
Hi @frgfm, streaming a dataset that contains a TAR file requires some tweaks because (contrary to ZIP files), tha TAR archive does not allow random access to any of the contained member files. Instead they have to be accessed sequentially (in the order in which they were put into the TAR file when created) and yielded.

So when iterating over the TAR file content, when an image file is found, we need to yield it (and not keeping it in memory, which will require huge RAM memory for large datasets). But when yielding an image file, we also need to yield with it what we call "metadata": the class label, and other textual information (for example, for audio files, sometimes we also add info such as the speaker ID, their sex, their age,...).

All this information usually is stored in what we call the metadata file: either a JSON or a CSV/TSV file.

But if this is also inside the TAR archive, we need to find this file in the first place when iterating the TAR archive, so that we already have this information when we find an image file and we can yield the image file and its metadata info.

Therefore:
- either the TAR archive contains the metadata file as the first member when iterating it (something we cannot change as it is done at the creation of the TAR file)
- or if not, then we need to have the metadata file elsewhere
  - in these cases, what we do (if the dataset license allows it) is:
    - we download the TAR file locally, we extract the metadata file and we host the metadata on the Hub
    - we modify the dataset loading script so that it first downloads the metadata file (and reads it) and only then starts iterating the content of the TAR archive file

See an example of this process we recently did for "google/fleurs" (their metadata files for "train" were at the end of the TAR archives, after all audio files): https://huggingface.co/datasets/google/fleurs/discussions/4
- we uploaded the metadata file to the Hub
- we adapted the loading script to use it
Issue: Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Comment:
Hi @albertvillanova :wave: 

Thanks, since my last message, I went through the repo of https://huggingface.co/datasets/food101/blob/main/food101.py and managed to get it to work in the end :pray: 

Here it is: https://huggingface.co/datasets/frgfm/imagenette

I appreciate you opening an issue to document the process, it might help a few!
Issue: Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Comment:
Great to see that you manage to make your dataset streamable. :rocket: 

I'm closing this issue, as for the docs update there is another issue opened:
- #4711
Issue: Cannot load LinCE dataset

Comment:
Hi @finiteautomata, thanks for reporting.

Unfortunately, I'm not able to reproduce your issue:
```python
In [1]: from datasets import load_dataset
   ...: dataset = load_dataset("lince", "ner_spaeng")
Downloading builder script: 20.8kB [00:00, 9.09MB/s]                                                                                                                                                       
Downloading metadata: 31.2kB [00:00, 13.5MB/s]                                                                                                                                                             
Downloading and preparing dataset lince/ner_spaeng (download: 2.93 MiB, generated: 18.45 MiB, post-processed: Unknown size, total: 21.38 MiB) to .../.cache/huggingface/datasets/lince/ner_spaeng/1.0.0/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.08M/3.08M [00:01<00:00, 2.73MB/s]
Dataset lince downloaded and prepared to .../.cache/huggingface/datasets/lince/ner_spaeng/1.0.0/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 630.66it/s]

In [2]: dataset
Out[2]: 
DatasetDict({
    train: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 33611
    })
    validation: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 10085
    })
    test: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 23527
    })
})
``` 

Please note that for this dataset, the original data files are not hosted on the Hugging Face Hub, but on https://ritual.uh.edu
And sometimes, the server might be temporarily unavailable, as your error message said (trying to connect to the server timed out):
```
ConnectionError: Couldn't reach https://ritual.uh.edu/lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='ritual.uh.edu', port=443): Max retries exceeded with url: /lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7feb1c45a690>, 'Connection to ritual.uh.edu timed out. (connect timeout=100)'))")))
```
In these cases you could:
- either contact the owners of the data server where the data is hosted to inform them about the issue in their server
- or re-try after waiting some time: usually these issues are just temporary
Issue: Distributed data parallel training for streaming datasets

Comment:
Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)


 EDIT: here is a code example 

```python
# ds = ds.with_format("torch")
# dataloader = DataLoader(ds, num_workers=num_workers)
```

EDIT: `with_format("torch")` is not required, now you can just do
```python
dataloader = DataLoader(ds, num_workers=num_workers)
```
Issue: Distributed data parallel training for streaming datasets

Comment:
@cyk1337 does streaming datasets with multi-gpu works for you? I am testing on one node with multiple gpus, but this is freezing, https://github.com/huggingface/datasets/issues/5123 
In case you could make this work, could you share with me your data-loading codes?
thank you
Issue: Distributed data parallel training for streaming datasets

Comment:
This has been implemented in `datasets` 2.8:
```python
from datasets.distributed import split_dataset_by_node

ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

docs: https://huggingface.co/docs/datasets/use_with_pytorch#distributed
Issue: Distributed data parallel training for streaming datasets

Comment:
i'm having hanging issues with this when using DDP and allocating the datasets with `split_dataset_by_node` 🤔

--- 
### edit
I don't want to pollute this thread, but for the sake of following up, I observed hanging close to the final iteration of the dataloader. I think this was happening on the final shard. First, I removed the final shard and things worked. Then (including all shards), I reordered the list of shards: `load_dataset('json', data_files=reordered, streaming=True)` and no hang. 

I won't open an issue yet bc I am not quite sure about this observation.
Issue: Dataset Viewer issue for rajistics/indian_food_images

Comment:
Hi, thanks for reporting. I triggered a refresh of the preview for this dataset, and it works now. I'm not sure what occurred.
<img width="1019" alt="Capture d’écran 2022-07-18 à 11 01 52" src="https://user-images.githubusercontent.com/1676121/179541327-f62ecd5e-a18a-4d91-b316-9e2ebde77a28.png">
Issue: How to assign new values to Dataset?

Comment:
Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.
Issue: How to assign new values to Dataset?

Comment:
Hi! I tried your proposed solution, but it does not solve my problem unfortunately. I am working with a set of protein sequences that have been tokenized with ESM, but some sequences are longer than `max_length`, they have been truncated in the tokenization. So now I want to truncate my labels as well, but that does not work with a mapping (e.g. `dset.map` as you suggested). Specifically, what I did was the following:

```
def postprocess_tokenize(tokenized_data):
    """
    adjust label lengths if they dont match.
    """
    if len(tokenized_data['input_ids']) < len(tokenized_data['labels']):
        new_labels = tokenized_data['labels'][:len(tokenized_data['input_ids'])]
        tokenized_data["labels"] = new_labels
    return tokenized_data

tokenized_data = tokenized_data.map(postprocess_tokenize, batched=True) # this does not adjust the labels...
```

Any tips on how to do this properly?

More generally, I am wondering why the DataCollator supports padding but does not support truncation? Seems odd to me.

Thanks in advance!
Issue: IndexError when loading ImageFolder

Comment:
Hi, thanks for reporting! If there are no examples in ImageFolder, the `label` column is of type `ClassLabel(names=[])`, which leads to an error in [this line](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/arrow_writer.py#L387) as `asdict(info)` calls `Features({..., "label": {'num_classes': 0, 'names': [], 'id': None, '_type': 'ClassLabel'}})`, which then calls `require_decoding` [here](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/features/features.py#L1516) on the dict value it does not expect.

I see two ways to fix this:
* custom `asdict` where `dict_factory` is also applied on the `dict` object itself besides dataclasses (the built-in implementation calls `type(dict_obj)` - this means we also need to fix `Features.to_dict` btw) 
* implement `DatasetInfo.to_dict` (though adding `to_dict` to a data class is a bit weird IMO)

@lhoestq Which one of these approaches do you like more?
Issue: IndexError when loading ImageFolder

Comment:
Small pref for the first option, it feels weird to know that `Features()` can be called with a dictionary of types defined as dictionaries instead of type instances.
Issue: Dataset Viewer issue for codeparrot/xlcost-text-to-code

Comment:
There seems to be an issue with the `C++-snippet-level` config:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("codeparrot/xlcost-text-to-code", "C++-snippet-level")
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 352, in get_dataset_config_info
    info.splits = {
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

I remove the dataset-viewer tag since it's not directly related.

Pinging @huggingface/datasets
Issue: Dataset Viewer issue for codeparrot/xlcost-text-to-code

Comment:
Thanks I found that this subset wasn't properly defined the the config, I fixed it. Now I can see the subsets but I get this error for the viewer
````
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```
Issue: Dataset Viewer issue for codeparrot/xlcost-text-to-code

Comment:
Yes, the cache is being refreshed, hopefully, it will work in some minutes for all the splits. Some are already here:

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code/viewer/Python-snippet-level/train

<img width="1533" alt="Capture d’écran 2022-07-18 à 12 04 06" src="https://user-images.githubusercontent.com/1676121/179553933-64d874fa-ada9-4b82-900e-082619523c20.png">
Issue: Cant pass streaming dataset to dataloader after take()

Comment:
Hi! Calling `take` on an iterable/streamable dataset makes it not possible to shard the dataset, which in turn disables multi-process loading (attempts to split the workload over the shards), so to go past this limitation, you can either use single-process loading in `DataLoader` (`num_workers=None`) or fetch the first `50_000/batch_size` batches in the loop.
Issue: Random 400 Client Error when pushing dataset

Comment:
I am having the same issue. Even the simple example from the documentation gives me the 400 Error


> from datasets import load_dataset
> 
> dataset = load_dataset("stevhliu/demo")
> dataset.push_to_hub("processed_demo")


`requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/REDACTED/commit/main (Request ID: e-tPnYTiCdB5KPmSL86dQ)`

I "fixed" it by initializing a new virtual environment with only datasets==2.5.2 installed.

The workaround consists of saving to disk then loading from disk and pushing to hub but from the new clean virtual environment.
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
Are you able to reproduce this? My example is small enough that it should be easy to try.
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
Hi! Thanks for reporting and providing a reproducible example. Indeed, by default, `datasets` performs an expensive cast on the values returned by `map` to convert them to one of the types supported by PyArrow (the underlying storage format used by `datasets`). This cast is not needed on NumPy arrays as PyArrow supports them natively, so one way to make this transform faster is to add `return_tensors="np"` to the tokenizer call. 

I think we should mention this in the docs (cc @stevhliu)
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
I tested this tokenize function and indeed noticed a casting. However it seems to only concerns the `offset_mapping` field, which contains a list of tuples, that is converted to a list of lists. Since `pyarrow` also supports tuples, we actually don't need to convert the tuples to lists. 

I think this can be changed here: 

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/src/datasets/features/features.py#L382-L383

```diff
- if isinstance(obj, list): 
+ if isinstance(obj, (list, tuple)): 
```

and here: 

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/src/datasets/features/features.py#L386-L387

```diff
- return obj if isinstance(obj, list) else [], isinstance(obj, tuple)
+ return obj, False
```

@srobertjames can you try applying these changes and let us know if it helps ? If so, feel free to open a Pull Request to contribute this improvement if you want :)
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
Wow, adding `return_tensors="np"` sped up my example by a **factor 17x** of and completely eliminated the casting! I'd recommend not only to document it, but to make that the default.

The code at https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb does not specify `return_tensors="np"` but yet avoids the casting penalty.  How does it do that?  (The ntbk seems to do `return_overflowing_tokens=True, return_offsets_mapping=True,`).

Also, surprisingly enough, using `return_tensors="pt"` (which is my eventual application) yields this error:
```
TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types 
[<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>]. 
When using `batched=True`, make sure provided `function` returns a `dict` of types like 
`(<class 'list'>, <class 'numpy.ndarray'>)`.
```
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
Setting the output to `"np"` makes the whole pipeline fast because it moves the data buffers from rust to python to arrow using zero-copy, and also because it does eliminate the casting completely ;)

Have you had a chance to try eliminating the tuple casting using the trick above ?
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
@lhoestq I just benchmarked the two edits to `features.py` above, and they appear to solve the problem, bringing my original example to within 20% the speed of the output `"np"` example.  Nice!

For a pull request, do you suggest simply following https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md ?
Issue: Dataset.map gets stuck on _cast_to_python_objects

Comment:
Cool ! Sure feel free to follow these instructions to open a PR :) thanks !
Issue: Unable to use dataset with PyTorch dataloader

Comment:
Hi! `para_crawl` has a single column of type `Translation`, which stores translation dictionaries. These dictionaries can be stored in a NumPy array but not in a PyTorch tensor since PyTorch only supports numeric types. In `datasets`, the conversion to `torch` works as follows: 
1. convert PyArrow table to NumPy arrays 
2. convert NumPy arrays to Torch tensors. 

The 2nd step is problematic for your case as `datasets` attempts to convert the array of dictionaries to a PyTorch tensor. One way to fix this is to use the [preprocessing logic](https://github.com/huggingface/transformers/blob/8581a798c0a48fca07b29ce2ca2ef55adcae8c7e/examples/pytorch/translation/run_translation.py#L440-L458) from the Transformers translation script. And on our side, I think we can replace a NumPy array of dicts with a dict of NumPy array if the feature type is `Translation`/`TranslationVariableLanguages` (one array for each language) to get the official PyTorch error message for strings in such case.
Issue: Issue loading datasets -- pyarrow.lib has no attribute

Comment:
Hi @margotwagner, thanks for reporting.

Unfortunately, I'm not able to reproduce your bug: in an environment with datasets-2.3.2 and pyarrow-8.0.0, I can load the datasets without any problem:
```python
>>> ds = load_dataset("glue", "cola")
>>> ds
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 8551
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1043
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1063
    })
})

>>> import pyarrow
>>> pyarrow.__version__
8.0.0
>>> from pyarrow.lib import IpcReadOptions
>>> IpcReadOptions
pyarrow.lib.IpcReadOptions
```

I think you may have a problem in your Python environment: maybe you have also an old version of pyarrow that has precedence when importing it.

Could you please check this (just after you tried to load the dataset and got the error)?
```python
>>> import pyarrow
>>> pyarrow.__version__
```
Issue: load_datasets on csv returns everything as a string 

Comment:
Hi @courtneysprouse, thanks for reporting.

Yes, you are right: by default the "csv" loader loads all columns as strings. 

You could tweak this behavior by passing the `feature` argument to `load_dataset`, but it is also true that currently it is not possible to perform some kind of casts, due to lacking of implementation in PyArrow. For example:
```python
import datasets

features = datasets.Features(
    {
        "tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(datasets.Value("int32")),
    }
)

new_conll = datasets.load_dataset("csv", data_files="ner_conll.csv", features=features)
```
gives `ArrowNotImplementedError` error:
```
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from string to list using function cast_list
```

On the other hand, if you just would like to save and afterwards load your dataset, you could use `save_to_disk` and `load_from_disk` instead. These functions preserve all data types.
```python
>>> orig_conll.save_to_disk("ner_conll")

>>> from datasets import load_from_disk

>>> new_conll = load_from_disk("ner_conll")
>>> new_conll
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 14042
    })
    validation: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3251
    })
    test: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3454
    })
})
>>> new_conll["train"][0]
{'chunk_tags': [11, 21, 11, 12, 21, 22, 11, 12, 0],
 'id': '0',
 'ner_tags': [3, 0, 7, 0, 0, 0, 7, 0, 0],
 'pos_tags': [22, 42, 16, 21, 35, 37, 16, 21, 7],
 'tokens': ['EU',
  'rejects',
  'German',
  'call',
  'to',
  'boycott',
  'British',
  'lamb',
  '.']}
>>> new_conll["train"].features
{'chunk_tags': Sequence(feature=ClassLabel(num_classes=23, names=['O', 'B-ADJP', 'I-ADJP', 'B-ADVP', 'I-ADVP', 'B-CONJP', 'I-CONJP', 'B-INTJ', 'I-INTJ', 'B-LST', 'I-LST', 'B-NP', 'I-NP', 'B-PP', 'I-PP', 'B-PRT', 'I-PRT', 'B-SBAR', 'I-SBAR', 'B-UCP', 'I-UCP', 'B-VP', 'I-VP'], id=None), length=-1, id=None),
 'id': Value(dtype='string', id=None),
 'ner_tags': Sequence(feature=ClassLabel(num_classes=9, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'], id=None), length=-1, id=None),
 'pos_tags': Sequence(feature=ClassLabel(num_classes=47, names=['"', "''", '#', '$', '(', ')', ',', '.', ':', '``', 'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB'], id=None), length=-1, id=None),
 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```