helmo/github-issues huggingface dataset:
Fix Authentification Error while streaming
State: closed | Author: hkjeon13 (NONE)

I fixed a few errors when it occurs while streaming the private dataset on the Huggingface Hub.
```
from datasets import load_dataset

dataset = load_dataset(<repo_id>, use_auth_token=<private_token>, streaming=True)
for d in dataset['train']:
    print(d)
    break # this is for checking
```
This code is an example for streaming private datasets. 
when the version of the datasets is 2.2.2, it works well but datasets>2.2.2 occurs error like this,
```
/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py in raise_for_status(self)
1007 status=self.status,
1008 message=self.reason,
→ 1009 headers=self.headers,
1010 )
1011

ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/.../train-00000-of-00001-168b451062c67c34.parquet')
```
(this is an example on the dataset has `parquet` extenstion)
It seems that the `xisfile `module in `download/streaming_download_manager.py` couldn't recognize the file on "https://huggingface.co/~".

so I add three lines.
With this change, there is no error anymore(but this code is ad-hoc).

Comments: Hi, thanks for working on this, but the fix for this has already been merged in https://github.com/huggingface/datasets/pull/4608.
Enable streaming dataset to use the "all" split
State: closed | Author: cakiki (CONTRIBUTOR)

Fixes #4637

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4698). All of your documentation changes will be reflected on that endpoint.

@albertvillanova 
Adding the validation split causes these two `assert_called_once` assertions to fail with `AssertionError: Expected 'ArrowWriter' to have been called once. Called 2 times`:

https://github.com/huggingface/datasets/blob/main/tests/test_builder.py#L548-L562

It might be better to create a new dummy generator for the streaming tests, WDYT? Alternatively we could test for `self.call_count` equalling 2.

@cakiki have you read my comment in the issue page?
https://github.com/huggingface/datasets/issues/4637#issuecomment-1175984812

Streaming with `split=all` seems to be working, will fix the failing test next

Not sure if marking the PR as "ready for review" actually notified you, so tagging @albertvillanova just in case :smiley_cat: 

cc @lhoestq 

Hi @cakiki, still interested in working on this? :) 

@albertvillanova So sorry; I have no idea how this slipped through the cracks. Yes, I'd still like to work on this. Is it okay if I DM you on slack?

Sure!! And nevermind!
Add MANtIS dataset
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)
Labels: name: dataset contribution

This PR adds MANtIS dataset.
Arxiv: [https://arxiv.org/abs/1912.04639](https://arxiv.org/abs/1912.04639)
Github: [https://github.com/Guzpenha/MANtIS](https://github.com/Guzpenha/MANtIS)

README and dataset tags are WIP.

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks for your contribution, @bhavitvyamalik. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest that you create this dataset there. Please, feel free to tell us if you need some help.
Distributed data parallel training for streaming datasets
State: open | Author: cyk1337 (NONE)
Labels: name: enhancement

### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training? 

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?

Comments: Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)


 EDIT: here is a code example 

```python
# ds = ds.with_format("torch")
# dataloader = DataLoader(ds, num_workers=num_workers)
```

EDIT: `with_format("torch")` is not required, now you can just do
```python
dataloader = DataLoader(ds, num_workers=num_workers)
```

@cyk1337 does streaming datasets with multi-gpu works for you? I am testing on one node with multiple gpus, but this is freezing, https://github.com/huggingface/datasets/issues/5123 
In case you could make this work, could you share with me your data-loading codes?
thank you

+1

This has been implemented in `datasets` 2.8:
```python
from datasets.distributed import split_dataset_by_node

ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

docs: https://huggingface.co/docs/datasets/use_with_pytorch#distributed

i'm having hanging issues with this when using DDP and allocating the datasets with `split_dataset_by_node` 🤔

--- 
### edit
I don't want to pollute this thread, but for the sake of following up, I observed hanging close to the final iteration of the dataloader. I think this was happening on the final shard. First, I removed the final shard and things worked. Then (including all shards), I reordered the list of shards: `load_dataset('json', data_files=reordered, streaming=True)` and no hang. 

I won't open an issue yet bc I am not quite sure about this observation.

@wconnell would you mind opening a different bug issue and giving more details?
https://github.com/huggingface/datasets/issues/new?assignees=&labels=&template=bug-report.yml

Thanks.
Dataset Viewer issue for rajistics/indian_food_images
State: closed | Author: rajshah4 (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/rajistics/indian_food_images/viewer/rajistics--indian_food_images/train

### Description

I have a train/test split in my dataset 
<img width="410" alt="Screen Shot 2022-07-15 at 11 44 42 AM" src="https://user-images.githubusercontent.com/6808012/179293215-7b419ec3-3527-46f2-8dad-adbc5568cfa0.png">
t
The dataset viewer works for the test split (images of indian food), but does not show my train split.  My guess is maybe there is some corrupt image file that is guessing this.  But I have no idea.

The original dataset was pulled from here: https://www.kaggle.com/datasets/l33tc0d3r/indian-food-classification?resource=download-directory


### Owner

Yes

Comments: Hi, thanks for reporting. I triggered a refresh of the preview for this dataset, and it works now. I'm not sure what occurred.
<img width="1019" alt="Capture d’écran 2022-07-18 à 11 01 52" src="https://user-images.githubusercontent.com/1676121/179541327-f62ecd5e-a18a-4d91-b316-9e2ebde77a28.png">
Refactor base extractors
State: closed | Author: albertvillanova (MEMBER)

This PR:
- Refactors base extractors as subclasses of `BaseExtractor`:
  - this is an abstract class defining the interface with:
    - `is_extractable`: abstract class method
    - `extract`: abstract static method
- Implements abstract `MagicNumberBaseExtractor` (as subclass of `BaseExtractor`): 
  - this has a default implementation of `is_extractable`
  - this improves performance (reducing the number of file reads) by allowing passing already read `magic_number`
- Refactors `Extractor`:
  - reads magic number from file only once

This PR deprecates:
```python
is_extractable, extractor = self.extractor.is_extractable(input_path, return_extractor=True)
self.extractor.extract(input_path, output_path, extractor=extractor)
```
and uses more Pythonic instead:
```python
extractor_format = self.extractor.infer_extractor_format(input_path)
self.extractor.extract(input_path, output_path, extractor_format)
```

Comments: _The documentation is not available anymore as the PR was closed or merged._
Skip test_extractor only for zstd param if zstandard not installed
State: closed | Author: albertvillanova (MEMBER)

Currently, if `zstandard` is not installed, `test_extractor` is skipped for all compression format parameters.

This PR fixes `test_extractor` so that if `zstandard` is not installed, `test_extractor` is skipped only for the `zstd` compression parameter, that is, it is not skipped for all the other compression parameters (`gzip`, `xz`,...).

Comments: _The documentation is not available anymore as the PR was closed or merged._
Align logging with Transformers (again)
State: closed | Author: mariosasko (COLLABORATOR)

Fix #2832 

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4686). All of your documentation changes will be reflected on that endpoint.

I wasn't aware of https://github.com/huggingface/datasets/pull/1845 before opening this PR. This issue seems much more complex now ...
How to assign new values to Dataset?
State: closed | Author: beyondguo (NONE)
Labels: name: enhancement

![image](https://user-images.githubusercontent.com/37113676/179149159-bbbda0c8-a661-403c-87ed-dc2b4219cd68.png)

Hi, if I want to change some values of the dataset, or add new columns to it, how can I do it?

For example, I want to change all the labels of the SST2 dataset to `0`:
```python
from datasets import load_dataset
data = load_dataset('glue','sst2')

data['train']['label'] = [0]*len(data)
```

I will get the error:
```
TypeError: 'Dataset' object does not support item assignment
```

Comments: Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.

Hi! I tried your proposed solution, but it does not solve my problem unfortunately. I am working with a set of protein sequences that have been tokenized with ESM, but some sequences are longer than `max_length`, they have been truncated in the tokenization. So now I want to truncate my labels as well, but that does not work with a mapping (e.g. `dset.map` as you suggested). Specifically, what I did was the following:

```
def postprocess_tokenize(tokenized_data):
    """
    adjust label lengths if they dont match.
    """
    if len(tokenized_data['input_ids']) < len(tokenized_data['labels']):
        new_labels = tokenized_data['labels'][:len(tokenized_data['input_ids'])]
        tokenized_data["labels"] = new_labels
    return tokenized_data

tokenized_data = tokenized_data.map(postprocess_tokenize, batched=True) # this does not adjust the labels...
```

Any tips on how to do this properly?

More generally, I am wondering why the DataCollator supports padding but does not support truncation? Seems odd to me.

Thanks in advance!
Update create dataset card docs
State: closed | Author: stevhliu (MEMBER)
Labels: name: documentation

This PR proposes removing the [online dataset card creator](https://huggingface.co/datasets/card-creator/) in favor of simply copy/pasting a template and using the [Datasets Tagger app](https://huggingface.co/spaces/huggingface/datasets-tagging) to generate the tags. The Tagger app provides more guidance by showing all possible values a user can select in the dropdown menus, whereas the online dataset card creator doesn't, which can make it difficult to know what tag values to input.

Let me know what you think! :)

Comments: _The documentation is not available anymore as the PR was closed or merged._
weird issue/bug with columns (dataset iterable/stream mode)
State: open | Author: eunseojo (CONTRIBUTOR)

I have a dataset online (CloverSearch/cc-news-mutlilingual) that has a bunch of columns, two of which are "score_title_maintext" and "score_title_description". the original files are jsonl formatted. I was trying to iterate through via streaming mode and grab all "score_title_description" values, but I kept getting key not found after a certain point of iteration. I found that some json objects in the file don't have "score_title_description". And in SOME cases, this returns a NONE and in others it just gets a key error. Why is there an inconsistency here and how can I fix it?
IndexError when loading ImageFolder
State: closed | Author: johko (NONE)
Labels: name: bug

## Describe the bug
Loading an image dataset with `imagefolder` throws `IndexError: list index out of range` when the given folder contains a non-image file (like a csv).

## Steps to reproduce the bug
Put a csv file in a folder with images and load it:
```python
import datasets
 datasets.load_dataset("imagefolder", data_dir=path/to/folder)
```

## Expected results
I would expect a better error message, like `Unsupported file` or even the dataset loader just ignoring every file that is not an image in that case.

## Actual results
Here is the whole traceback:

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.11.0-051100-generic-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Comments: Hi, thanks for reporting! If there are no examples in ImageFolder, the `label` column is of type `ClassLabel(names=[])`, which leads to an error in [this line](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/arrow_writer.py#L387) as `asdict(info)` calls `Features({..., "label": {'num_classes': 0, 'names': [], 'id': None, '_type': 'ClassLabel'}})`, which then calls `require_decoding` [here](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/features/features.py#L1516) on the dict value it does not expect.

I see two ways to fix this:
* custom `asdict` where `dict_factory` is also applied on the `dict` object itself besides dataclasses (the built-in implementation calls `type(dict_obj)` - this means we also need to fix `Features.to_dict` btw) 
* implement `DatasetInfo.to_dict` (though adding `to_dict` to a data class is a bit weird IMO)

@lhoestq Which one of these approaches do you like more?


Small pref for the first option, it feels weird to know that `Features()` can be called with a dictionary of types defined as dictionaries instead of type instances.
Dataset Viewer issue for codeparrot/xlcost-text-to-code
State: closed | Author: loubnabnl (NONE)

### Link

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code

### Description

Error
```
Server Error
Status code:   400
Exception:     TypeError
Message:       'NoneType' object is not iterable
```
Before I did a minor change in the dataset script (removing some comments), the viewer was working but not properely, it wasn't showing the dataset subsets. But the data can be loaded successfully.

Thanks!

### Owner

Yes

Comments: There seems to be an issue with the `C++-snippet-level` config:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("codeparrot/xlcost-text-to-code", "C++-snippet-level")
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 352, in get_dataset_config_info
    info.splits = {
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

I remove the dataset-viewer tag since it's not directly related.

Pinging @huggingface/datasets 

Thanks I found that this subset wasn't properly defined the the config, I fixed it. Now I can see the subsets but I get this error for the viewer
````
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

Yes, the cache is being refreshed, hopefully, it will work in some minutes for all the splits. Some are already here:

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code/viewer/Python-snippet-level/train

<img width="1533" alt="Capture d’écran 2022-07-18 à 12 04 06" src="https://user-images.githubusercontent.com/1676121/179553933-64d874fa-ada9-4b82-900e-082619523c20.png">


I think all the splits are working as expected now

Perfect, thank you!
Cant pass streaming dataset to dataloader after take()
State: open | Author: zankner (NONE)
Labels: name: bug

## Describe the bug
I am trying to pass a streaming version of c4 to a dataloader, but it can't be passed after I call `dataset.take(n)`. Some functions such as `shuffle()` can be applied without breaking the dataloader but not take.

## Steps to reproduce the bug
```python
import datasets
import torch

dset = datasets.load_dataset(path='c4', name='en', split="train", streaming=True)
dset = dset.take(50_000)
dset = dset.with_format("torch")

num_workers = 8
batch_size = 512

loader = torch.utils.data.DataLoader(dataset=dset,
                                     batch_size=batch_size,
                                     num_workers=num_workers)
for batch in loader:
    ...
```

## Expected results
No error thrown when iterating over the dataloader

## Actual results
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/root/.local/lib/python3.9/site-packages/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py", line 48, in __iter__
    for key, example in self._iter_shard(shard_idx):
  File "/root/.local/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 586, in _iter_shard
    yield from ex_iterable.shard_data_sources(shard_idx)
  File "/root/.local/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 60, in shard_data_sources
    raise NotImplementedError(f"{type(self)} doesn't implement shard_data_sources yet")
NotImplementedError: <class 'datasets.iterable_dataset.TakeExamplesIterable'> doesn't implement shard_data_sources yet

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Comments: Hi! Calling `take` on an iterable/streamable dataset makes it not possible to shard the dataset, which in turn disables multi-process loading (attempts to split the workload over the shards), so to go past this limitation, you can either use single-process loading in `DataLoader` (`num_workers=None`) or fetch the first `50_000/batch_size` batches in the loop.
Unable to use dataset with PyTorch dataloader
State: open | Author: BlueskyFR (NONE)
Labels: name: bug

## Describe the bug

When using `.with_format("torch")`, an arrow table is returned and I am unable to use it by passing it to a PyTorch DataLoader: please see the code below.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torch.utils.data import DataLoader

ds = load_dataset(
    "para_crawl",
    name="enfr",
    cache_dir="/tmp/test/",
    split="train",
    keep_in_memory=True,
)

dataloader = DataLoader(ds.with_format("torch"), num_workers=32)
print(next(iter(dataloader)))
```

Is there something I am doing wrong? The documentation does not say much about the behavior of `.with_format()` so I feel like I am a bit stuck here :-/

Thanks in advance for your help!

## Expected results

The code should run with no error

## Actual results

```
AttributeError: 'str' object has no attribute 'dtype'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.18.0-348.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Comments: Hi! `para_crawl` has a single column of type `Translation`, which stores translation dictionaries. These dictionaries can be stored in a NumPy array but not in a PyTorch tensor since PyTorch only supports numeric types. In `datasets`, the conversion to `torch` works as follows: 
1. convert PyArrow table to NumPy arrays 
2. convert NumPy arrays to Torch tensors. 

The 2nd step is problematic for your case as `datasets` attempts to convert the array of dictionaries to a PyTorch tensor. One way to fix this is to use the [preprocessing logic](https://github.com/huggingface/transformers/blob/8581a798c0a48fca07b29ce2ca2ef55adcae8c7e/examples/pytorch/translation/run_translation.py#L440-L458) from the Transformers translation script. And on our side, I think we can replace a NumPy array of dicts with a dict of NumPy array if the feature type is `Translation`/`TranslationVariableLanguages` (one array for each language) to get the official PyTorch error message for strings in such case.
Issue loading datasets -- pyarrow.lib has no attribute
State: closed | Author: margotwagner (NONE)
Labels: name: bug

## Describe the bug
I am trying to load sentiment analysis datasets from huggingface, but any dataset I try to use via load_dataset, I get the same error:
`AttributeError: module 'pyarrow.lib' has no attribute 'IpcReadOptions'`

## Steps to reproduce the bug
```python
dataset = load_dataset("glue", "cola")
```

## Expected results
Download datasets without issue.

## Actual results
`AttributeError: module 'pyarrow.lib' has no attribute 'IpcReadOptions'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 8.0.0
- Pandas version: 1.1.0


Comments: Hi @margotwagner, thanks for reporting.

Unfortunately, I'm not able to reproduce your bug: in an environment with datasets-2.3.2 and pyarrow-8.0.0, I can load the datasets without any problem:
```python
>>> ds = load_dataset("glue", "cola")
>>> ds
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 8551
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1043
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1063
    })
})

>>> import pyarrow
>>> pyarrow.__version__
8.0.0
>>> from pyarrow.lib import IpcReadOptions
>>> IpcReadOptions
pyarrow.lib.IpcReadOptions
```

I think you may have a problem in your Python environment: maybe you have also an old version of pyarrow that has precedence when importing it.

Could you please check this (just after you tried to load the dataset and got the error)?
```python
>>> import pyarrow
>>> pyarrow.__version__
```