Minimartzz/github-issues huggingface dataset:
Fix require torchaudio and refactor test requirements
State: closed

Currently there is a bug in `require_torchaudio` (indeed it is requiring `sox` instead):
```python
def require_torchaudio(test_case):
    if find_spec("sox") is None:
...
```

The bug was introduced by:
- #3685
  - Commit: https://github.com/huggingface/datasets/pull/3685/commits/b5a3e7122d49c4dcc9333b1d8d18a833fc04b940

which moved
```python
require_sndfile = pytest.mark.skipif(
    # In Windows and OS X, soundfile installs sndfile
    (sys.platform != "linux" and find_spec("soundfile") is None)
    # In Linux, soundfile throws RuntimeError if sndfile not installed with distribution package manager
    or (sys.platform == "linux" and find_library("sndfile") is None),
    reason="Test requires 'sndfile': `pip install soundfile`; "
    "Linux requires sndfile installed with distribution package manager, e.g.: `sudo apt-get install libsndfile1`",
)
require_sox = pytest.mark.skipif(
    find_library("sox") is None,
    reason="Test requires 'sox'; only available in non-Windows, e.g.: `sudo apt-get install sox`",
)
require_torchaudio = pytest.mark.skipif(find_spec("torchaudio") is None, reason="Test requires 'torchaudio'")
```
to
```python
def require_sndfile(test_case):
    """
    Decorator marking a test that requires soundfile.
    These tests are skipped when soundfile isn't installed.
    """
    if (sys.platform != "linux" and find_spec("soundfile") is None) or (
        sys.platform == "linux" and find_library("sndfile") is None
    ):
        test_case = unittest.skip(
            "test requires 'sndfile': `pip install soundfile`; "
            "Linux requires sndfile installed with distribution package manager, e.g.: `sudo apt-get install libsndfile1`",
        )(test_case)
    return test_case


def require_sox(test_case):
    """
    Decorator marking a test that requires sox.
    These tests are skipped when sox isn't installed.
    """
    if find_library("sox") is None:
        return unittest.skip("test requires 'sox'; only available in non-Windows, e.g.: `sudo apt-get install sox`")(
            test_case
        )
    return test_case


def require_torchaudio(test_case):
    """
    Decorator marking a test that requires torchaudio.
    These tests are skipped when torchaudio isn't installed.
    """
    if find_spec("sox") is None:
        return unittest.skip("test requires 'torchaudio'")(test_case)
    return test_case
```

This PR;
- fixes the bug in `require_torchaudio`
- refactors the test requirements back to using `pytest` instead of `unittest`
  - the text in `pytest.skipif` `reason` can be used if needed in a test body: `require_torchaudio.kwargs["reason"]`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Dataset Viewer issue for TheNoob3131/mosquito-data
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

_No response_

### Description

Getting this error when trying to view dataset preview: 
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/TheNoob3131/mosquito-data/resolve/8aceebd6c4a359d216d10ef020868bd9e8c986dd/0_Africa_train.csv')


### Owner

_No response_

Comments:
Thanks for reporting. I refreshed the dataset viewer and it now works as expected.

https://huggingface.co/datasets/TheNoob3131/mosquito-data

<img width="1135" alt="Capture d’écran 2022-07-18 à 13 15 22" src="https://user-images.githubusercontent.com/1676121/179566497-e47f1a27-fd84-4a8d-9d7f-2e0f2da803df.png">

We will investigate why it occurred in the first place


By chance, could you provide some details about the operations done on the dataset: was it private? gated?

Yes, it was a private dataset, and when I made it public, the Dataset Preview did not work. 

However, now when I make the dataset private, it says that the Dataset Preview has been disabled. Why is this?

Thanks for the details. For now, the dataset viewer is always disabled on private datasets (see https://huggingface.co/docs/hub/datasets-viewer for more details)

Hi, it was working fine for a few hours, but then I can't see the dataset viewer again (public dataset). Why is this still happening?
It's the same error too:
![image](https://user-images.githubusercontent.com/53668030/179602465-f220f971-d3aa-49ba-a31b-60510f4c2a89.png)


OK? This is a bug, thanks for help spotting and reproducing it (it occurs when a dataset is switched to private, then to public). We will be working on it, meanwhile, I've restored the dataset viewer manually again.
Fix empty examples in xtreme dataset for bucc18 config
State: closed

As reported in https://huggingface.co/muibk, there are empty examples in xtreme/bucc18.de

I applied your fix @mustaszewski

I also used a dict to make the dataset generation much faster

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I guess the report link is this instead: https://huggingface.co/datasets/xtreme/discussions/1
Fix crd3
State: closed

As reported in https://huggingface.co/datasets/crd3/discussions/1#62cc377073b2512b81662794, each split of the dataset was containing the same data. This issues comes from a bug in the dataset script

I fixed it and also uploaded the data to hf.co to make the dataset work in streaming mode

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip tests only for lz4/zstd params if not installed
State: closed

Currently, if `zstandard` or `lz4` are not installed, `test_compression_filesystems` and `test_streaming_dl_manager_extract_all_supported_single_file_compression_types` are skipped for all compression format parameters.

This PR fixes these tests, so that if `zstandard` or `lz4` are not installed, the tests are skipped only for the corresponding compression parameters (`zstd` or `lz4`), whereas the tests are not skipped for all the other compression parameters (`gzip`, `xz` and `bz2`).

Related to:
- #4688

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make cast in `from_pandas` more robust
State: closed

Make the cast in `from_pandas` more robust (as it was done for the packaged modules in https://github.com/huggingface/datasets/pull/4364)

This should be useful in situations like [this one](https://discuss.huggingface.co/t/loading-custom-audio-dataset-and-fine-tuning-model/8836/4).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix Authentification Error while streaming
State: closed

I fixed a few errors when it occurs while streaming the private dataset on the Huggingface Hub.
```
from datasets import load_dataset

dataset = load_dataset(<repo_id>, use_auth_token=<private_token>, streaming=True)
for d in dataset['train']:
    print(d)
    break # this is for checking
```
This code is an example for streaming private datasets. 
when the version of the datasets is 2.2.2, it works well but datasets>2.2.2 occurs error like this,
```
/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py in raise_for_status(self)
1007 status=self.status,
1008 message=self.reason,
→ 1009 headers=self.headers,
1010 )
1011

ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/.../train-00000-of-00001-168b451062c67c34.parquet')
```
(this is an example on the dataset has `parquet` extenstion)
It seems that the `xisfile `module in `download/streaming_download_manager.py` couldn't recognize the file on "https://huggingface.co/~".

so I add three lines.
With this change, there is no error anymore(but this code is ad-hoc).

Comments:
Hi, thanks for working on this, but the fix for this has already been merged in https://github.com/huggingface/datasets/pull/4608.
Enable streaming dataset to use the "all" split
State: closed

Fixes #4637

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4698). All of your documentation changes will be reflected on that endpoint.

@albertvillanova 
Adding the validation split causes these two `assert_called_once` assertions to fail with `AssertionError: Expected 'ArrowWriter' to have been called once. Called 2 times`:

https://github.com/huggingface/datasets/blob/main/tests/test_builder.py#L548-L562

It might be better to create a new dummy generator for the streaming tests, WDYT? Alternatively we could test for `self.call_count` equalling 2.

@cakiki have you read my comment in the issue page?
https://github.com/huggingface/datasets/issues/4637#issuecomment-1175984812

Streaming with `split=all` seems to be working, will fix the failing test next

Not sure if marking the PR as "ready for review" actually notified you, so tagging @albertvillanova just in case :smiley_cat: 

cc @lhoestq 

Hi @cakiki, still interested in working on this? :) 

@albertvillanova So sorry; I have no idea how this slipped through the cracks. Yes, I'd still like to work on this. Is it okay if I DM you on slack?

Sure!! And nevermind!
Add MANtIS dataset
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution

This PR adds MANtIS dataset.
Arxiv: [https://arxiv.org/abs/1912.04639](https://arxiv.org/abs/1912.04639)
Github: [https://github.com/Guzpenha/MANtIS](https://github.com/Guzpenha/MANtIS)

README and dataset tags are WIP.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your contribution, @bhavitvyamalik. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest that you create this dataset there. Please, feel free to tell us if you need some help.
Distributed data parallel training for streaming datasets
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training? 

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?

Comments:
Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)


 EDIT: here is a code example 

```python
# ds = ds.with_format("torch")
# dataloader = DataLoader(ds, num_workers=num_workers)
```

EDIT: `with_format("torch")` is not required, now you can just do
```python
dataloader = DataLoader(ds, num_workers=num_workers)
```

@cyk1337 does streaming datasets with multi-gpu works for you? I am testing on one node with multiple gpus, but this is freezing, https://github.com/huggingface/datasets/issues/5123 
In case you could make this work, could you share with me your data-loading codes?
thank you

+1

This has been implemented in `datasets` 2.8:
```python
from datasets.distributed import split_dataset_by_node

ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

docs: https://huggingface.co/docs/datasets/use_with_pytorch#distributed

i'm having hanging issues with this when using DDP and allocating the datasets with `split_dataset_by_node` 🤔

--- 
### edit
I don't want to pollute this thread, but for the sake of following up, I observed hanging close to the final iteration of the dataloader. I think this was happening on the final shard. First, I removed the final shard and things worked. Then (including all shards), I reordered the list of shards: `load_dataset('json', data_files=reordered, streaming=True)` and no hang. 

I won't open an issue yet bc I am not quite sure about this observation.

@wconnell would you mind opening a different bug issue and giving more details?
https://github.com/huggingface/datasets/issues/new?assignees=&labels=&template=bug-report.yml

Thanks.
Dataset Viewer issue for rajistics/indian_food_images
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/rajistics/indian_food_images/viewer/rajistics--indian_food_images/train

### Description

I have a train/test split in my dataset 
<img width="410" alt="Screen Shot 2022-07-15 at 11 44 42 AM" src="https://user-images.githubusercontent.com/6808012/179293215-7b419ec3-3527-46f2-8dad-adbc5568cfa0.png">
t
The dataset viewer works for the test split (images of indian food), but does not show my train split.  My guess is maybe there is some corrupt image file that is guessing this.  But I have no idea.

The original dataset was pulled from here: https://www.kaggle.com/datasets/l33tc0d3r/indian-food-classification?resource=download-directory


### Owner

Yes

Comments:
Hi, thanks for reporting. I triggered a refresh of the preview for this dataset, and it works now. I'm not sure what occurred.
<img width="1019" alt="Capture d’écran 2022-07-18 à 11 01 52" src="https://user-images.githubusercontent.com/1676121/179541327-f62ecd5e-a18a-4d91-b316-9e2ebde77a28.png">
Refactor base extractors
State: closed

This PR:
- Refactors base extractors as subclasses of `BaseExtractor`:
  - this is an abstract class defining the interface with:
    - `is_extractable`: abstract class method
    - `extract`: abstract static method
- Implements abstract `MagicNumberBaseExtractor` (as subclass of `BaseExtractor`): 
  - this has a default implementation of `is_extractable`
  - this improves performance (reducing the number of file reads) by allowing passing already read `magic_number`
- Refactors `Extractor`:
  - reads magic number from file only once

This PR deprecates:
```python
is_extractable, extractor = self.extractor.is_extractable(input_path, return_extractor=True)
self.extractor.extract(input_path, output_path, extractor=extractor)
```
and uses more Pythonic instead:
```python
extractor_format = self.extractor.infer_extractor_format(input_path)
self.extractor.extract(input_path, output_path, extractor_format)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip test_extractor only for zstd param if zstandard not installed
State: closed

Currently, if `zstandard` is not installed, `test_extractor` is skipped for all compression format parameters.

This PR fixes `test_extractor` so that if `zstandard` is not installed, `test_extractor` is skipped only for the `zstd` compression parameter, that is, it is not skipped for all the other compression parameters (`gzip`, `xz`,...).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Align logging with Transformers (again)
State: closed

Fix #2832 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4686). All of your documentation changes will be reflected on that endpoint.

I wasn't aware of https://github.com/huggingface/datasets/pull/1845 before opening this PR. This issue seems much more complex now ...
How to assign new values to Dataset?
State: closed
Labels: description: New feature or request
name: enhancement

![image](https://user-images.githubusercontent.com/37113676/179149159-bbbda0c8-a661-403c-87ed-dc2b4219cd68.png)

Hi, if I want to change some values of the dataset, or add new columns to it, how can I do it?

For example, I want to change all the labels of the SST2 dataset to `0`:
```python
from datasets import load_dataset
data = load_dataset('glue','sst2')

data['train']['label'] = [0]*len(data)
```

I will get the error:
```
TypeError: 'Dataset' object does not support item assignment
```

Comments:
Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.

Hi! I tried your proposed solution, but it does not solve my problem unfortunately. I am working with a set of protein sequences that have been tokenized with ESM, but some sequences are longer than `max_length`, they have been truncated in the tokenization. So now I want to truncate my labels as well, but that does not work with a mapping (e.g. `dset.map` as you suggested). Specifically, what I did was the following:

```
def postprocess_tokenize(tokenized_data):
    """
    adjust label lengths if they dont match.
    """
    if len(tokenized_data['input_ids']) < len(tokenized_data['labels']):
        new_labels = tokenized_data['labels'][:len(tokenized_data['input_ids'])]
        tokenized_data["labels"] = new_labels
    return tokenized_data

tokenized_data = tokenized_data.map(postprocess_tokenize, batched=True) # this does not adjust the labels...
```

Any tips on how to do this properly?

More generally, I am wondering why the DataCollator supports padding but does not support truncation? Seems odd to me.

Thanks in advance!
Update create dataset card docs
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

This PR proposes removing the [online dataset card creator](https://huggingface.co/datasets/card-creator/) in favor of simply copy/pasting a template and using the [Datasets Tagger app](https://huggingface.co/spaces/huggingface/datasets-tagging) to generate the tags. The Tagger app provides more guidance by showing all possible values a user can select in the dropdown menus, whereas the online dataset card creator doesn't, which can make it difficult to know what tag values to input.

Let me know what you think! :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
weird issue/bug with columns (dataset iterable/stream mode)
State: open

I have a dataset online (CloverSearch/cc-news-mutlilingual) that has a bunch of columns, two of which are "score_title_maintext" and "score_title_description". the original files are jsonl formatted. I was trying to iterate through via streaming mode and grab all "score_title_description" values, but I kept getting key not found after a certain point of iteration. I found that some json objects in the file don't have "score_title_description". And in SOME cases, this returns a NONE and in others it just gets a key error. Why is there an inconsistency here and how can I fix it?

Comments:
IndexError when loading ImageFolder
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Loading an image dataset with `imagefolder` throws `IndexError: list index out of range` when the given folder contains a non-image file (like a csv).

## Steps to reproduce the bug
Put a csv file in a folder with images and load it:
```python
import datasets
 datasets.load_dataset("imagefolder", data_dir=path/to/folder)
```

## Expected results
I would expect a better error message, like `Unsupported file` or even the dataset loader just ignoring every file that is not an image in that case.

## Actual results
Here is the whole traceback:

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.11.0-051100-generic-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Comments:
Hi, thanks for reporting! If there are no examples in ImageFolder, the `label` column is of type `ClassLabel(names=[])`, which leads to an error in [this line](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/arrow_writer.py#L387) as `asdict(info)` calls `Features({..., "label": {'num_classes': 0, 'names': [], 'id': None, '_type': 'ClassLabel'}})`, which then calls `require_decoding` [here](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/features/features.py#L1516) on the dict value it does not expect.

I see two ways to fix this:
* custom `asdict` where `dict_factory` is also applied on the `dict` object itself besides dataclasses (the built-in implementation calls `type(dict_obj)` - this means we also need to fix `Features.to_dict` btw) 
* implement `DatasetInfo.to_dict` (though adding `to_dict` to a data class is a bit weird IMO)

@lhoestq Which one of these approaches do you like more?


Small pref for the first option, it feels weird to know that `Features()` can be called with a dictionary of types defined as dictionaries instead of type instances.
Dataset Viewer issue for codeparrot/xlcost-text-to-code
State: closed

### Link

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code

### Description

Error
```
Server Error
Status code:   400
Exception:     TypeError
Message:       'NoneType' object is not iterable
```
Before I did a minor change in the dataset script (removing some comments), the viewer was working but not properely, it wasn't showing the dataset subsets. But the data can be loaded successfully.

Thanks!

### Owner

Yes

Comments:
There seems to be an issue with the `C++-snippet-level` config:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("codeparrot/xlcost-text-to-code", "C++-snippet-level")
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 352, in get_dataset_config_info
    info.splits = {
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

I remove the dataset-viewer tag since it's not directly related.

Pinging @huggingface/datasets 

Thanks I found that this subset wasn't properly defined the the config, I fixed it. Now I can see the subsets but I get this error for the viewer
````
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

Yes, the cache is being refreshed, hopefully, it will work in some minutes for all the splits. Some are already here:

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code/viewer/Python-snippet-level/train

<img width="1533" alt="Capture d’écran 2022-07-18 à 12 04 06" src="https://user-images.githubusercontent.com/1676121/179553933-64d874fa-ada9-4b82-900e-082619523c20.png">


I think all the splits are working as expected now

Perfect, thank you!