noamaanMulla-03/datasets-issues huggingface dataset:
Handling unlabeled datasets

Hi!

Is there a way for datasets to produce unlabeled instances (e.g., the `ClassLabel` can be nullable).

For example, I want to use the MNLI dataset reader ( https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py ) on a file that doesn't have the `gold_label` field. I tried setting `"label": data.get("gold_label")`, but got the following error:

```
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 953, in encode_example
    return encode_nested_example(self, example)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in encode_nested_example
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 875, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 653, in encode_example
    if not -1 <= example_data < self.num_classes:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```

What's the proper way to handle reading unlabeled datasets, especially for downstream usage with Transformers?

Labels: name: enhancement

Hi @nelson-liu,

You can pass the parameter `features` to `load_dataset`: https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset

If you look at the code of the MNLI script you referred in your question (https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py#L62-L77), you can see how the Features were originally specified. 

Feel free to use it as a template, customize it and pass it to `load_dataset` using the parameter `features`.

ah got it, thanks!
`datasets.keyhash.DuplicatedKeysError` for `drop` and `adversarial_qa/adversarialQA`

## Describe the bug
Failure to generate the datasets (`drop` and subset `adversarialQA` from `adversarial_qa`) because of duplicate keys.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("drop")
load_dataset("adversarial_qa", "adversarialQA")
```

## Expected results
The examples keys should be unique.

## Actual results
```bash
>>> load_dataset("drop")
Using custom data configuration default
Downloading and preparing dataset drop/default (download: 7.92 MiB, generated: 111.88 MiB, post-processed: Unknown size, total: 119.80 MiB) to /home/hf/.cache/huggingface/datasets/drop/default/0.1.0/7a94f1e2bb26c4b5c75f89857c06982967d7416e5af935a9374b9bccf5068026...
Traceback (most recent call last):         
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 992, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/arrow_writer.py", line 409, in finalize
    self.check_duplicate_keys()
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 28553293-d719-441b-8f00-ce3dc6df5398
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1044-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0


Labels: name: bug

very much related: https://github.com/huggingface/datasets/pull/2333

Hi @VictorSanh, thank you for reporting this issue with duplicated keys.

- The issue with "adversarial_qa" was fixed 23 days ago: #2433. Current version of `datasets` (1.8.0) includes the patch.
- I am investigating the issue with `drop`. I'll ping you to keep you informed.

Hi @VictorSanh, the issue is already fixed and merged into master branch and will be included in our next release version 1.9.0.

thank you!
Add support for Split.ALL

The title says it all.


Honestly, I think we should fix some other issues in Split API before this change. E. g. currently the following will not work, even though it should:
```python
import datasets
datasets.load_dataset("sst", split=datasets.Split.TRAIN+datasets.Split.TEST)  # AssertionError
```

EDIT:
Actually, think it's OK to merge this PR because the fix will not touch this PR's code.
Some tests hang on Windows

Currently, several tests hang on Windows if the max path limit of 260 characters is not disabled. This happens due to the changes introduced by #2223 that cause an infinite loop in `WindowsFileLock` described in #2220.  This can be very tricky to debug, so I think now is a good time to address these issues/PRs. IMO throwing an error is too harsh, but maybe we can emit a warning in the top-level `__init__.py ` on startup if long paths are not enabled.


Labels: name: bug

Hi ! That would be nice indeed to at least have a warning, since we don't handle the max path length limit.
Also if we could have an error instead of an infinite loop I'm sure windows users will appreciate that

Unfortunately, I know this problem very well... 😅 

I remember having proposed to throw an error instead of hanging in an infinite loop #2220: 60c7d1b6b71469599a27147a08100f594e7a3f84, 8c8ab60018b00463edf1eca500e434ff061546fc 
but @lhoestq told me:
> Note that the filelock module comes from this project that hasn't changed in years - while still being used by ten of thousands of projects:
https://github.com/benediktschmitt/py-filelock
> 
> Unless we have proper tests for this, I wouldn't recommend to change it

I opened an Issue requesting a warning/error at startup for that case: #2224

@albertvillanova Thanks for additional info on this issue.

Yes, I think the best option is to throw an error instead of suppressing it in a loop. I've considered 2 more options, but I don't really like them:
1. create a temporary file with a filename longer than 255 characters on import; if this fails, long paths are not enabled and raise a warning. I'm not sure about this approach because I don't like the idea of creating a temporary file on import for this purpose.
2. check if long paths are enabled with [this code](https://stackoverflow.com/a/46546731/14095927). As mentioned in the comment, this code relies on an undocumented function and Win10-specific.
Make numpy arrow extractor faster

I changed the NumpyArrowExtractor to call directly to_numpy and see if it can lead to speed-ups as discussed in https://github.com/huggingface/datasets/issues/2498

This could make the numpy/torch/tf/jax formatting faster


Looks like we have a nice speed up in some benchmarks. For example:
- `read_formatted numpy 5000`:  4.584777 sec -> 0.487113 sec
- `read_formatted torch 5000`:  4.565676 sec -> 1.289514 sec

Can we convert this draft to PR @lhoestq ?

Ready for review ! cc @vblagoje

@lhoestq I tried the branch and it works for me. Although performance trace now shows a speedup, the overall pre-training speed up is minimal. But that's on my plate to explore further. 

Thanks for investigating @vblagoje 

@albertvillanova , do you have any comments on this PR ? Otherwise I think we can merge it
Raise FileNotFoundError in WindowsFileLock

Closes #2443 


Hi ! Could you clarify what it fixes exactly and give more details please ? Especially why this is related to the windows hanging error ?

This has already been merged, but I'll clarify the idea of this PR. Before this merge, FileLock was the only component affected by the max path limit on Windows (that came to my notice) because of its infinite loop that would suppress errors. So instead of suppressing the `FileNotFoundError` that is thrown by `os.open` if the file name is longer than the max allowed path length, this PR reraises it to notify the user.
Fix FileSystems documentation

### What this fixes:
This PR resolves several issues I discovered in the documentation on the `datasets.filesystems` module ([this page](https://huggingface.co/docs/datasets/filesystems.html)).

### What were the issues?
When I originally tried implementing the code examples I faced several bugs attributed to:

- out of date [botocore](https://github.com/boto/botocore) call signatures
- capitalization errors in the `S3FileSystem` class name (written as `S3Filesystem` in one place)
- call signature errors for the `S3FileSystem` class constructor (uses parameter `sessions` instead of `session` in some places) (see [`s3fs`](https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem) for where this constructor signature is defined)

### Testing/reviewing notes
Instructions for generating the documentation locally: [here](https://github.com/huggingface/datasets/tree/master/docs#generating-the-documentation).
update discofuse link cc @ekQ

Updating the discofuse link: https://github.com/google-research-datasets/discofuse/commit/fd4b120cb3dd19a417e7f3b5432010b574b5eeee


The CI is failing because the dataset tags for `discofuse` are missing. I'm merging this PR since this is unrelated to this PR, but feel free to open another PR to add the tags here if you have some time:

https://github.com/huggingface/datasets/blob/19408f9fab85c79b966085574cd2da3b90959179/datasets/discofuse/README.md#L1-L5

The missing tags are:
```
'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'pretty_name', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
```
Thanks again !
Missing ClassLabel encoding in Json loader

Currently if you want to load a json dataset this way
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then if your features has ClassLabel types and if your json data needs class label encoding (i.e. if the labels in the json files are strings and not integers), then it would fail:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ArrowInvalid: Failed to parse string: 'O' as a scalar of type int64
```

This is because it just tries to cast the string data to integers, without applying the mapping str->int first

The current workaround is to do instead
```python
dataset = load_dataset("json", data_files=data_files)
dataset = dataset.map(features.encode_example, features=features)
```

Labels: name: bug
load_dataset("web_nlg") NonMatchingChecksumError

Hi! It seems the WebNLG dataset gives a NonMatchingChecksumError.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset('web_nlg', name="release_v3.0_en", split="dev")
```

Gives

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://gitlab.com/shimorina/webnlg-dataset/-/archive/master/webnlg-dataset-master.zip']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: macOS-11.3.1-x86_64-i386-64bit
- Python version: 3.9.4
- PyArrow version: 3.0.0

Also tested on Linux, with python 3.6.8

Labels: name: bug

Hi ! Thanks for reporting. This is due to the WebNLG repository that got updated today.
I just pushed a fix at #2558 - this shouldn't happen anymore in the future.

This is fixed on `master` now :)
We'll do a new release soon !
fix Dataset.map when num_procs > num rows

closes #2470

## Testing notes
To run updated tests:
```sh
pytest tests/test_arrow_dataset.py -k "BaseDatasetTest and test_map_multiprocessing" -s
```
With Python code (to view warning):
```python
from datasets import Dataset


dataset = Dataset.from_dict({"x": ["sample"]})
print(len(dataset))
dataset.map(lambda x: x, num_proc=10)
```


Hi ! Thanks for fixing this :)

Looks like you have tons of changes due to code formatting.
We're using `black` for this, with a custom line length. To run our code formatting, you just need to run
```
make style
```

Then for the windows error in the CI, I'm looking into it. It's probably just a file that isn't properly closed

CI is all green now ! Thanks :)

There are still many code formatting changes in your PR - probably due to the first commit you did.
To avoid conflicts with future PRs it would be nice to only have the changes related to the `num_proc` warning, and not have all those code formatting changes,

Could you try remove those code formatting changes ?

If it's easier for you, you can make a new branch from `master` if needed

Thanks, @lhoestq! Apologies for the half-baked commits yesterday! I wasn’t able to step back in to resolve those CI issues until this morning.

Also, I’m surprised that `make style` isn’t resolving the formatting changes. I’m a bit stumped on that, so I’m going to re-apply on a new branch and open a PR as you suggested.
[Timit_asr] Make sure not only the first sample is used 

When playing around with timit I noticed that only the first sample is used for all indices. I corrected this typo so that the dataset is correctly loaded.


cc @lhoestq  @vrindaprabhu

Failing `run (push)` is unrelated -> merging

Thanks for fixing this, it was affecting my runs for https://github.com/huggingface/transformers/pull/10581/

I am seeing this very late! Sorry for the blunder everyone! :(
Add Parquet loader + from_parquet and to_parquet

Continuation of #2247 

I added a "parquet" dataset builder, as well as the methods `Dataset.from_parquet` and `Dataset.to_parquet`.
As usual, the data are converted to arrow in a batched way to avoid loading everything in memory.


`pyarrow` 1.0.0 doesn't support some types in parquet, we'll have to bump its minimum version.

Also I still need to add dummy data to test the parquet builder.

I had to bump the minimum pyarrow version to 3.0.0 to properly support parquet.

Everything is ready for review now :)
I reused pretty much the same tests we had for CSV

Done !
Now we're still allowing pyarrow>=1.0.0, but when users want to use parquet features they're asked to update to pyarrow>=3.0.0
Weights of model checkpoint not initialized for RobertaModel for Bertscore

When applying bertscore out of the box, 

```Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']```

Following the typical usage from https://huggingface.co/docs/datasets/loading_metrics.html

```
from datasets import load_metric
metric = load_metric('bertscore')

# Example of typical usage
for batch in dataset:
    inputs, references = batch
    predictions = model(inputs)
    metric.add_batch(predictions=predictions, references=references)
score = metric.compute(lang="en")
#score = metric.compute(model_type="roberta-large") # gives the same error
```

I am concerned about this because my usage shouldn't require any further fine-tuning and most people would expect to use BertScore out of the box? I realised the huggingface code is a wrapper around https://github.com/Tiiiger/bert_score, but I think this repo is anyway relying on the model code and weights from huggingface repo.... 

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1041-aws-x86_64-with-glibc2.27
- Python version:  3.9.5
- PyArrow version: 3.0.0


Labels: name: bug

Hi @suzyahyah, thanks for reporting.

The message you get is indeed not an error message, but a warning coming from Hugging Face `transformers`. The complete warning message is:
```
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

In this case, this behavior IS expected and you can safely ignore the warning message.

The reason is that you are just using RoBERTa to get the contextual embeddings of the input sentences/tokens, thus leaving away its head layer, whose weights are ignored.

Feel free to reopen this issue if you need further explanations.

Hi @suzyahyah, I have created a Pull Request to filter out that warning message in this specific case, since the behavior is as expected and the warning message can only cause confusion for users (as in your case).
fix Dataset.map when num_procs > num rows

closes #2470

## Testing notes
To run updated tests:
```sh
pytest tests/test_arrow_dataset.py -k "BaseDatasetTest and test_map_multiprocessing" -s
```
With Python code (to view warning):
```python
from datasets import Dataset


dataset = Dataset.from_dict({"x": ["sample"]})
print(len(dataset))
dataset.map(lambda x: x, num_proc=10)
```
Add interleave_datasets for map-style datasets

### Add interleave_datasets for map-style datasets

Add support for map-style datasets (i.e. `Dataset` objects) in `interleave_datasets`.
It was only supporting iterable datasets (i.e. `IterableDataset` objects).

### Implementation details

It works by concatenating the datasets and then re-order the indices to make the new dataset.

### TODO
- [x] tests
- [x] docs

Close #2563
Finding right block-size with JSON loading difficult for user

As reported by @thomwolf, while loading a JSON Lines file with "json" loading script, he gets
> json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 383)


Labels: name: bug

This was actually a second error arising from a too small block-size in the json reader.

Finding the right block size is difficult for the layman user
Add mC4

AllenAI is now hosting the processed C4 and mC4 dataset in this repo: https://huggingface.co/datasets/allenai/c4
Thanks a lot to them !

In this PR I added the mC4 dataset builder. It supports 108 languages

You can load it with
```python
from datasets import load_dataset

en_mc4 = load_dataset("mc4", "en")
fr_mc4 = load_dataset("mc4", "fr")
en_and_fr_mc4 = load_dataset("mc4", languages=["en", "fr"])
```

It also supports streaming, if you don't want to download hundreds of GB of data:
```python
en_mc4 = load_dataset("mc4", "en", streaming=True)
```

Regarding the dataset_infos.json, I will add them once I have them.

Also we can work on the dataset card at that will be at https://huggingface.co/datasets/mc4
For now I just added a link to https://huggingface.co/datasets/allenai/c4 as well as a few sections
Add C4

## Adding a Dataset
- **Name:** *C4*
- **Description:** *https://github.com/allenai/allennlp/discussions/5056*
- **Paper:** *https://arxiv.org/abs/1910.10683*
- **Data:** *https://huggingface.co/datasets/allenai/c4*
- **Motivation:** *Used a lot for pretraining*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Should fix https://github.com/huggingface/datasets/issues/1710

Labels: name: dataset request

Update on this: I'm computing the checksums of the data files. It will be available soon

Added in #2575 :)
Inject templates for ASR datasets

This PR adds ASR templates for 5 of the most common speech datasets on the Hub, where "common" is defined by the number of models trained on them.

I also fixed a bunch of the tags in the READMEs 😎 


Wait until #2567 is merged so we can benefit from the tagger :)

thanks for the feedback @lhoestq! i've added the new language codes and this PR should be ready for a merge :)
Add skip and take

As discussed in https://github.com/huggingface/datasets/pull/2375#discussion_r657084544 I added the `IterableDataset.skip` and `IterableDataset.take` methods that allows to do basic splitting of iterable datasets.

You can create new dataset with the first `n` examples using `IterableDataset.take()`, or you can get a dataset with the rest of the examples by skipping the first `n` examples with `IterableDataset.skip()`

One implementation detail:

Using `take` (or `skip`) prevents future dataset shuffling from shuffling the dataset shards, otherwise the taken examples could come from other shards. In this case it only uses the shuffle buffer.
I would have loved to allow the shards of the taken examples to be shuffled anyway, but since we don't know in advance the length of each shard we don't know what shards to take or skip.
I think this is ok though since users can shuffle before doing take or skip. I mentioned this in the documentation

cc @vblagoje @lewtun 


@lhoestq looks good. I tried with https://huggingface.co/datasets/vblagoje/wikipedia_snippets_streamed and it worked nicely. I would add more unit tests for edge cases. What happens if the n is larger than the total number of samples? Just to make sure these cases are handled properly. 

Yup I'll add the tests thanks ;)

Moreover, I just noticed something in your wiki snippets code. FYI you're using `++passage_counter ` at https://huggingface.co/datasets/vblagoje/wikipedia_snippets_streamed/blob/main/wikipedia_snippets_streamed.py#L102 but in python this doesn't increment the value @vblagoje 

Thanks @lhoestq - not easy to convert after 10+ years of Java
Support pandas 1.3.0 read_csv

Workaround for this issue in pandas 1.3.0 : https://github.com/pandas-dev/pandas/issues/42387

The csv reader raises an error:
```python
/usr/local/lib/python3.7/dist-packages/pandas/io/parsers/readers.py in _refine_defaults_read(dialect, delimiter, delim_whitespace, engine, sep, error_bad_lines, warn_bad_lines, on_bad_lines, names, prefix, defaults)
   1304 
   1305     if names is not lib.no_default and prefix is not lib.no_default:
-> 1306         raise ValueError("Specified named and prefix; you can only specify one.")
   1307 
   1308     kwds["names"] = None if names is lib.no_default else names

ValueError: Specified named and prefix; you can only specify one.
```