noamaanMulla-03/datasets-issues huggingface dataset:
Deprecate num_proc parameter in  DownloadManager.extract

fixes #5132 : Deprecated the `num_proc` parameter in `DownloadManager.extract` by passing `num_proc` parameter to `map_nested` .


_The documentation is not available anymore as the PR was closed or merged._

Hey @mariosasko . Can you please help me with why the tests keep failing.  I have reviewed the code changes multiple times but can't spot any mistakes. 

You can fix this failure by formatting your code with the `make style` command (run it from the root of the cloned repo).

hey @mariosasko ,i cant understand how to use the `make style` command .I searched for it on the internet but cant find any results. 
So i formatted the code using vs-code document formatter. Hope this helps.

`make style` runs the "style" target defined here: https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12

This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile





> `make style` runs the "style" target defined here:
> 
> https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12
> 
> This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile

Thanks! I will look into this :relaxed:
Memory explosion when trying to access 4d tensors in datasets cast to torch or np

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors. 

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):  
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map( 
        create_4d_tensor, 
        batched=False, 
        writer_batch_size=100, 
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")
    
    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Fix dependency on dill version

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana 

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385


_The documentation is not available anymore as the PR was closed or merged._
Fix iwslt2017

The files were moved to google drive, I hosted them on the Hub instead (ok according to the license)

I also updated the `datasets_infos.json`


_The documentation is not available anymore as the PR was closed or merged._

CI fails are just abut missing tags in the dataset card, merging !

FYI, 

The checksums have not been edited from the changes of .tgz to .zip files, and as a result a `ExpectedMoreDownloadedFiles` error occurs. Updating them in the `dataset_infos.json` should fix the error. 

Thanks for reporting and sorry for the delay, I opened https://huggingface.co/datasets/iwslt2017/discussions/2 to fix this
Add ffmpeg4 installation instructions in warnings

Adds instructions on how to install `ffmpeg=4` on Linux (relevant for Colab users).

Looks pretty ugly because I didn't find a way to check `ffmpeg` version from python (without `subprocess.call()`; `ctypes.util.find_library` doesn't work`), so the warning is raised on each decoding. Any suggestions on how to make it look nice are welcome!

This is how it looks on Colab:
![image](https://user-images.githubusercontent.com/16348744/198052412-d48018d1-4416-4aa5-9114-f7f9b4af031f.png)



_The documentation is not available anymore as the PR was closed or merged._

To make it warn only once, feel free to use a global counter in python - and if the warning has already been done, you don't do it again

> Added the same formatting for the error message :)

nice!! thank you! 

> Oh and regarding the warning counter, you can do it in another PR maybe ?

Yes, more warnings is better then no warnings.... I'll merge when the CI passes
Fix CI require beam

This PR:
- Fixes the CI `require_beam`: before it was requiring PyTorch instead
  ```python
  def require_beam(test_case):
      if not config.TORCH_AVAILABLE:
          test_case = unittest.skip("test requires PyTorch")(test_case)
      return test_case
  ```
- Fixes a missing `require_beam` in `test_beam_based_builder_download_and_prepare_as_parquet`
- Refactors `require_beam` to use `pytest` (`skipif`) instead


_The documentation is not available anymore as the PR was closed or merged._

I'm merging this PR because it is quite a trivial fix and this is required by:
- #5166
Dataset order is not deterministic with ZIP archives and `iter_files`

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```


Thanks for reporting ! The issue doesn't come from shuffling, but from `beans` row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses `dl_manager.iter_files` on ZIP archives and the file order doesn't seen to be deterministic and changes across machines

Thank you for noticing indeed!

This is still a bug, so I'd keep this one open if you don't mind ;)

Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L276) and [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L485) (e.g. fsspec's `LocalFileSystem.glob` relies on `os.scandir`, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to `load_dataset` similar to TFDS' [`shuffle_files`](https://www.tensorflow.org/datasets/determinism#determinism_when_reading) or sort only if the number of data files is small?

We already return the result sorted at the end of `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository` if I'm not mistaken

@lhoestq Oh, you are right. Feel free to ignore my comment.

I think the corresponding PR is ready to be merged :hugs: 

@albertvillanova Thanks for the fix!
Audiofolder creates empty Dataset if files same level as metadata

## Describe the bug
When audio files are at the same level as the metadata (`metadata.csv` or `metadata.jsonl` ), the `load_dataset` returns a `DatasetDict` with no rows but the correct columns.

https://github.com/huggingface/datasets/blob/1ea4d091b7a4b83a85b2eeb8df65115d39af3766/docs/source/audio_dataset.mdx?plain=1#L88

## Steps to reproduce the bug
`metadata.csv`:
```csv
file_name,duration,transcription
./2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav,10.768,hello
```


```python
>>> audio_dataset = load_dataset("audiofolder", data_dir="/audio-data/")
>>> audio_dataset
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

I've tried, with no success,:
- setting `split` to something else so I don't get a `DatasetDict`,
- removing the `./`,
- using `.jsonl`.


## Expected results
```
 Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 1
})
```

## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.13.0-1025-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.0



Labels: name: bug

name: good first issue

name: hacktoberfest

Hi! Unfortunately, I can't reproduce this behavior. Instead, I get `ValueError: audio at 2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav doesn't have metadata in /audio-data/metadata.csv`, which can be fixed by removing the `./` from the file name.

(Link to a Colab that tries to reproduce this behavior: https://colab.research.google.com/drive/1IhQzULYi0Van1xLrN_SddBX1JF7mLZZK?usp=sharing)

I think we can make the file name matching part more robust by replacing `file_name` with `os.path.normpath(file_name)`, to ignore "./" among other things, in these two places:
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L319
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L388

@mariosasko Some tests failed (see my PR). Any thoughts on that?

Yes, I mentioned the solution in my review.

I realized what I was doing wrong.

The documentation puts the files in a subfolder.
Once I have done that, it worked.

But l agree that this should be handled better if possible.
Make filename matching more robust

Fix  #5046


_The documentation is not available anymore as the PR was closed or merged._

> I think we should also modify one of the metadata files in the `folder_based_builder` tests to make sure "./" is ignored now in the `file_name`

@mariosasko what do you mean here? I'm not sure which metadata file I should modify here

You can modify this line for instance: https://github.com/huggingface/datasets/blob/2699593b33ee63d17aad2a2bfddedd38a8df57b8/tests/packaged_modules/test_folder_based_builder.py#L135
Raise ffmpeg warnings only once

Our warnings looks nice now.

`librosa` warning that was raised at each decoding: 
```
/usr/local/lib/python3.7/dist-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
```
is suppressed with `filterwarnings("ignore")` in a context manager. That means the first warning is also ignored (setting `filterwarnings("once")` didn't work!), so I added info that audioread is used for decoding to our message. Hope it's enough.


Tests failed at first because they used to check if the warning was raised at (each) decoding in `librosa` case but now we throw only one warning (at first decoding). I removed this check for warnings, do you think it's fine? 


_The documentation is not available anymore as the PR was closed or merged._
Loading an external NER dataset

I need to use huggingface datasets to load a custom dataset similar to conll2003 but with more entities and each the files contain only two columns: word and ner tag.

I tried this code snnipet that I found here as an answer to a similar issue:

from datasets import Dataset

INPUT_COLUMNS = "ID Text NER".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

train = Dataset.from_generator(read_conll, gen_kwargs={"file": "some_path"})   

But the following error happened:

ValueError: Please pass `features` or at least one example when writing data
Allow passing a subset of output features to Dataset.map

### Feature request

Currently, map does one of two things to the features (if I'm not mistaken):

* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this

However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.

### Motivation

To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings

Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```

The result is a float again, since after the mapping operation it forces the old datatypes back on the data. 

Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
    return {**tokenizer(r["text"]), "y": int(r["y"])}

data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
    format_data,
    features=Features({'y': Value(dtype="int64")}),
    remove_columns=["text"],
)
```

Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.


### Your contribution

I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.

Labels: name: enhancement
An example or recommendations for creating large image datasets?

I know that Apache Beam and `datasets` have [some connector utilities](https://huggingface.co/docs/datasets/beam). But it's a little unclear what we mean by "But if you want to run your own Beam pipeline with Dataflow, here is how:". What does that pipeline do? 

As a user, I was wondering if we have this support for creating large image datasets. If so, we should mention that [here](https://huggingface.co/docs/datasets/image_dataset). 

Cc @lhoestq 


The beam utilities allow to prepare a dataset as parquet in your cloud storage. From my perspective this CLI is not super easy to use, but we've been working on a new python API to prepare a dataset in your cloud storage:
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("c4", "en")
builder.download_and_prepapre("s3://my-bucket/c4", file_format="parquet")
```

And to use Beam you can do:
```python
beam_runner = ...  # one of "SparkRunner", "DataFlowRunner", "DirectRunner", etc.
beam_options = ...

builder.download_and_prepapre(
    "s3://my-bucket/c4",
    file_format="parquet",
    beam_runner=beam_runner,
    beam_options=beam_options
)
```

Though Beam can be used ONLY if there is a dataset script based on the `BeamBasedBuilder` right now - it doesn't work on an arbitrary dataset (see [wikipedia.py](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py) for example).

Thanks! 

Would be nice to have something similar for creating large image datasets.
Prefer split patterns from directories over split patterns from filenames

related to https://github.com/huggingface/datasets/issues/4895



_The documentation is not available anymore as the PR was closed or merged._

Can we merge this one since the issue this PR fixes was reported for the second time? I also think we don't need a test for this simple change.

@mariosasko sure! could you please approve it? 

Hi there @polinaeterna @mariosasko! I have installed 5.2.3.dev0, which should have this fix. Unfortunately, I am still getting the error:
`ValueError: Unknown split "validation". Should be one of ['train'].` When I call `load_dataset("csv", data_files=files, split=split)`

Any help would be greatly appreciated!
decode mp3 with librosa if torchaudio is > 0.12 as a temporary workaround 

`torchaudio>0.12` fails with decoding mp3 files if `ffmpeg<4`. currently we ask users to downgrade torchaudio, but sometimes it's not possible as torchaudio version is binded to torch version. as a temporary workaround we can decode mp3 with librosa (though it 60 times slower, at least it works)

another option would be to ask users to install the required version of `ffmpeg`, but is non-trivial on colab: it's not in apt packages in ubuntu 18 and `conda` is not preinstalled (with `conda` it would be easily installable)

- [x] decode with torchaudio anyway if the version of ffmpeg is correct? it's 60 times faster
- [x] tests 
- [x] DO NOT FORGET to get back all the tests

see https://github.com/huggingface/datasets/issues/4776 and https://github.com/huggingface/datasets/issues/3663#issuecomment-1225797165 (there is a Colab notebook to reproduce the error)


_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

@lhoestq 

> Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

I'm not sure here, because from the one hand, if `torchaudio` works - it works 60 times faster then `librosa`.
But from the other hand, we will get inconsistent behavior (=different results of decoding) for users of `torchaudio>=0.12`. 
I'd better go for using `librosa` only to avoid inconsistency then. wdyt?

It seems a bit too constraining to not allow users who have a working torchaudio 0.12 setup to not use it. 

If the issue is about avoiding silent errors if the decoding changes, maybe we can log which back-end is used ? It can even be a warning with performance suggestions ("you're using librosa but torchaudio 0.xx is recommended").

Note that users can still have a requirements.txt or whatever in their projects if they really want full reproducibility (and it's the bare minimum imo)

There are multiple possible back-ends so it's maybe not reasonable to only allow one back-end, especially since each back-end has installation constrains and there's no "best" back-end.

Woohoo all green ! Feel free to merge if it's all good for you :)
I need to read the custom dataset in conll format

I need to read the custom dataset in conll format


Labels: name: enhancement

Hi! We don't currently have a builder for parsing custom `conll` datasets, but I guess we could add one as a packaged module (similarly to what [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/dataset_builders/conll/conll_dataset_builder.py) did). @lhoestq @albertvillanova WDYT?

In the meantime, you can use `Dataset.from_generator` to create a dataset as follows:
```python
from datasets import Dataset

# 2009 version
INPUT_COLUMNS = "ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

# (optional) pass custom features with `features=Features(...)`
dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "path/to/conll/file"})   
``` 

I think we could add a dedicated builder if you think this format is general enough.




> I think we could add a dedicated builder if you think this format is general enough.

I think its functions are incomplete. It should have to_ Conll and from_ There are two methods of conll.
Loading an external dataset in a format similar to conll2003

I'm trying to load a custom dataset in a Dataset object, it's similar to conll2003 but with 2 columns only (word entity), I used the following script:

features = datasets.Features(
            {"tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=["B-PER", .... etc.]))}
        )


from datasets import Dataset

INPUT_COLUMNS = "tokens ner_tags".split(" ")

def read_conll(file):
    #all_labels = []
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
          if line:
            if line.startswith("-DOCSTART-") and example["tokens"] != []:
                print(idx, example)
                yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            elif line == "\n" or (line.startswith("-DOCSTART-") and example["tokens"] == []):
              continue
            else:
                row_cols = line.split(" ")
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "/content/new_train.txt"}, features = features)   

The following error happened:
[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <genexpr>(.0)
    285     for key in unique_values(itertools.chain(*dicts)):  # set merge all keys
    286         # Will raise KeyError if the dict don't have the same keys
--> 287         yield key, tuple(d[key] for d in dicts)
    288 
    

TypeError: tuple indices must be integers or slices, not str

What does this mean and what should I modify?