helmo/github-issues huggingface dataset:
Fix docs custom stable version
State: closed | Author: albertvillanova (MEMBER)

Currently docs default version is 1.5.0. This PR fixes this and sets the latest version instead.

Comments: I see that @lhoestq overlooked this PR with his commit 07e2b05. 😢 

I'm adding a script so that this issue does not happen again.


For the moment, the script only includes `update_custom_js`, but in a follow-up PR I will include all the required steps to make a package release.

I think we just need to clarify the release process in setup.py instead of adding a script that does the replacement

@lhoestq I really think we should implement a script that performs the release (instead of doing it manually as it is done now), as it is already the case in `transformers`. I will do it in a next PR.

For the moment, this PR includes one of the steps of the release script.
Add TimeDial
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

Dataset: https://github.com/google-research-datasets/TimeDial

To-Do: Update README.md and add YAML tags

Comments: Hi @lhoestq,
I've pushed the updated README and tags. Let me know if anything is missing/needs some improvement!

~PS. I don't know why it's not triggering the build~
Issue in timit_asr database
State: closed | Author: hrahamim (NONE)
Labels: name: bug

## Describe the bug
I am trying to load the timit_asr dataset however only the first record is shown (duplicated over all the rows).
I am using the next code line
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))

The above code result with the same sentence duplicated ten times.
It also happens when I use the dataset viewer at Streamlit .

## Steps to reproduce the bug
from datasets import load_dataset
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))
data = dataset.to_pandas()

# Sample code to reproduce the bug
```

## Expected results
table with different row information

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.4.1 (also occur in the latest version)
- Platform: Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 1.15.3 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Comments: This bug was fixed in #1995. Upgrading datasets to version 1.6 fixes the issue!

Indeed was a fixed bug.
Works on version 1.8
Thanks
cache_dir parameter for load_from_disk ?
State: closed | Author: chbensch (NONE)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
When using Google Colab big datasets can be an issue, as they won't fit on the VM's disk. Therefore mounting google drive could be a possible solution. Unfortunatly when loading my own dataset by using the _load_from_disk_ function, the data gets cached to the VM's disk:

`
from datasets import load_from_disk

myPreprocessedData = load_from_disk("/content/gdrive/MyDrive/ASR_data/myPreprocessedData")

`
I know that chaching on google drive could slow down learning. But at least it would run.

**Describe the solution you'd like**
Add cache_Dir parameter to the load_from_disk function.

**Describe alternatives you've considered**
It looks like you could write a custom loading script for the load_dataset function. But this seems to be much too complex for my use case. Is there perhaps a template here that uses the load_from_disk function?


Comments: Hi ! `load_from_disk` doesn't move the data. If you specify a local path to your mounted drive, then the dataset is going to be loaded directly from the arrow file in this directory. The cache files that result from `map` operations are also stored in the same directory by default.

However note than writing data to your google drive actually fills the VM's disk (see https://github.com/huggingface/datasets/issues/643)

Given that, I don't think that changing the cache directory changes anything.

Let me know what you think

Thanks for your answer! I am a little surprised since I just want to read the dataset.

After debugging a bit, I noticed that the VM’s disk fills up when the tables (generator) are converted to a list:

https://github.com/huggingface/datasets/blob/5ba149773d23369617563d752aca922081277ec2/src/datasets/table.py#L850

If I try to iterate through the table’s generator e.g.: 

`length = sum(1 for x in tables)`

the VM’s disk fills up as well.

I’m running out of Ideas 😄 

Indeed reading the data shouldn't increase the VM's disk. Not sure what google colab does under the hood for that to happen

Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636
Add Disfl-QA
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

Dataset: https://github.com/google-research-datasets/disfl-qa

To-Do: Update README.md and add YAML tags

Comments: Sounds great! It'll make things easier for the user while accessing the dataset. I'll make some changes to the current file then.

I've updated with the suggested changes. Updated the README, YAML tags as well (not sure of Size category tag as I couldn't pass the path of `dataset_infos.json` for this dataset)
Fix automatic generation of Zenodo DOI
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

After the last release of Datasets (1.8.0), the automatic generation of the Zenodo DOI failed: it appears in yellow as "Received", instead of in green as "Published".

I have contacted Zenodo support to fix this issue.

TODO:
- [x] Check with Zenodo to fix the issue
- [x] Check BibTeX entry is right

Comments: I have received a reply from Zenodo support:
> We are currently investigating and fixing this issue related to GitHub releases. As soon as we have solved it we will reach back to you.

Other repo maintainers had the same problem with Zenodo. 

There is an open issue on their GitHub repo: zenodo/zenodo#2181

I have received the following request from Zenodo support:
> Could you send us the link to the repository as well as the release tag?

My reply:
> Sure, here it is:
> - Link to the repository: https://github.com/huggingface/datasets
> - Link to the repository at the release tag: https://github.com/huggingface/datasets/releases/tag/1.8.0
> - Release tag: 1.8.0

Zenodo issue has been fixed. The 1.8.0 release DOI can be found here: https://zenodo.org/record/4946100#.YMd6vKj7RPY
Fix PermissionError on Windows when using tqdm >=4.50.0
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

See: https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111

```
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
```
adding masahaner dataset
State: closed | Author: dadelani (CONTRIBUTOR)

Adding Masakhane dataset https://github.com/masakhane-io/masakhane-ner 

@lhoestq , can you please review

Comments: Thank you for the review. 

Thanks a lot for the corrections and comments. 

I have resolved point 2. The make style still throws some errors, please see below

black --line-length 119 --target-version py36 tests src benchmarks datasets/**/*.py metrics
/bin/sh: 1: black: not found
Makefile:13: recipe for target 'style' failed
make: *** [style] Error 127

Can you help to resolve this?

Thank you very much @lhoestq for the help.
fix: adjusting indexing for the labels.
State: closed | Author: drugilsberg (CONTRIBUTOR)

The labels index were mismatching the actual ones used in the dataset. Specifically `0` is used  for `SUPPORTS` and `1` is used for `REFUTES`
After this change, the `README.md` now reflects the content of `dataset_infos.json`.

Signed-off-by: Matteo Manica <drugilsberg@gmail.com>

Comments: > Good catch ! Thanks for fixing it

My pleasure🙏
Merge DatasetDict and Dataset
State: open | Author: albertvillanova (MEMBER)
Labels: name: enhancement

name: generic discussion

As discussed in #2424 and #2437 (please see there for detailed conversation):
- It would be desirable to improve UX with respect the confusion between DatasetDict and Dataset.
- The difference between Dataset and DatasetDict is an additional abstraction complexity that confuses "typical" end users. 
- A user expects a "Dataset" (whatever it contains multiple or a single split) and maybe it could be interesting to try to simplify the user-facing API as much as possible to hide this complexity from the end user.

Here is a proposal for discussion and refined (and potential abandon if it's not good enough):
- let's consider that a DatasetDict is also a Dataset with the various split concatenated one after the other
- let's disallow the use of integers in split names (probably not a very big breaking change)
- when you index with integers you access the examples progressively in split after the other is finished (in a deterministic order)
- when you index with strings/split name you have the same behavior as now (full backward compat)
- let's then also have all the methods of a Dataset on the DatasetDict

The end goal would be to merge both Dataset and DatasetDict object in a single object that would be (pretty much totally) backward compatible with both.


There are a few things that we could discuss if we want to merge Dataset and DatasetDict:

1. what happens if you index by a string ? Does it return the column or the split ? We could disallow conflicts between column names and split names to avoid ambiguities. It can be surprising to be able to get a column or a split using the same indexing feature
   ```
   from datasets import load_dataset

   dataset = load_dataset(...)
   dataset["train"]
   dataset["input_ids"]
   ```
2. what happens when you iterate over the object ? I guess it should iterate over the examples as a Dataset object, but a DatasetDict used to iterate over the splits as they are the dictionary keys. This is a breaking change that we can discuss.

Moreover regarding your points:

- integers are not allowed as split names already
- it's definitely doable to have all the methods. Maybe some of them like train_test_split that is currently only available for Dataset can be tweaked to work for a split dataset


cc: @thomwolf @lhoestq 

Comments: Any update on this? @lhoestq 

Unless there is high demande I don't think we will end up implementing this. This is a lot of work with very few advantages
Support sliced list arrays in cast
State: closed | Author: lhoestq (MEMBER)

There is this issue in pyarrow:
```python
import pyarrow as pa

arr = pa.array([[i * 10] for i in range(4)])
arr.cast(pa.list_(pa.int32()))  # works

arr = arr.slice(1)
arr.cast(pa.list_(pa.int32()))  # fails
# ArrowNotImplementedError("Casting sliced lists (non-zero offset) not yet implemented")
```

However in `Dataset.cast` we slice tables to cast their types (it's memory intensive), so we have the same issue.
Because of this it is currently not possible to cast a Dataset with a Sequence feature type (unless the table is small enough to not be sliced).

In this PR I fixed this by resetting the offset of `pyarrow.ListArray` arrays to zero in the table before casting.
I used `pyarrow.compute.subtract` function to update the offsets of the ListArray.

cc @abhi1thakur @SBrandeis
`Proto_qa` hosting seems to be broken
State: closed | Author: VictorSanh (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
The hosting (on Github) of the `proto_qa` dataset seems broken. I haven't investigated more yet, just flagging it for now. 

@zaidalyafeai if you want to dive into it, I think it's just a matter of changing the links in `proto_qa.py`

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("proto_qa")
```

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 630, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/hf/.cache/huggingface/modules/datasets_modules/datasets/proto_qa/445346efaad5c5f200ecda4aa7f0fb50ff1b55edde3003be424a2112c3e8102e/proto_qa.py", line 131, in _split_generators
    train_fpath = dl_manager.download(_URLs[self.config.name]["train"])
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 199, in download
    num_proc=download_config.num_proc,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 195, in map_nested
    return function(data_struct)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 218, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/iesl/protoqa-data/master/data/train/protoqa_train.jsonl
```

Comments: @VictorSanh , I think @mariosasko is already working on it.
Revert default in-memory for small datasets
State: closed | Author: albertvillanova (MEMBER)
Labels: name: enhancement

Users are reporting issues and confusion about setting default in-memory to True for small datasets.

We see 2 clear use cases of Datasets:
- the "canonical" way, where you can work with very large datasets, as they are memory-mapped and cached (after every transformation)
- some edge cases (speed benchmarks, interactive/exploratory analysis,...), where default in-memory can explicitly be enabled, and no caching will be done

After discussing with @lhoestq we have agreed to:
- revert this feature (implemented in #2182)
- explain in the docs how to optimize speed/performance by setting default in-memory

cc: @stas00 https://github.com/huggingface/datasets/pull/2409#issuecomment-856210552

Comments: cc: @krandiash (pinged in reverted PR).
Add align_labels_with_mapping function
State: closed | Author: lewtun (MEMBER)

This PR adds a helper function to align the `label2id` mapping between a `datasets.Dataset` and a classifier (e.g. a transformer with a `PretrainedConfig.label2id` dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from `datasets`.

An example where this is needed is if we naively try to evaluate `microsoft/deberta-base-mnli` on `mnli` because the model config has the following mappings:

```python
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }
```

while the `mnli` dataset has the `contradiction` and `neutral` labels swapped:

```python
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}
```

As a result, we get a much lower accuracy during evaluation:

```python
from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))
```

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

```python
mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)
```

cc @thomwolf @lhoestq 

Comments: @lhoestq i think this is ready for another review 🙂 

@lhoestq thanks for the feedback - it's now integrated :) 

i also added a comment about sorting the input label IDs

Created the PR here: https://github.com/huggingface/datasets/pull/2510

> Thanks ! Looks all good now :)
> 
> We will also need to have the `DatasetDict.align_labels_with_mapping` method. Let me quickly add it

thanks a lot! i always forget about `DatasetDict` - will be happy when it's just one "dataset" object :)

So, there seems to be a problem with the function align_labels_with_mapping for models like this: https://huggingface.co/huggingface/distilbert-base-uncased-finetuned-mnli]. At least with this model, but perhaps also with others, the model.config.label2id values are of type str not int, which crashes said function. After manually converting the model.config.label2id values to int, the script runs smoothly.
Update version in xor_tydi_qa.py
State: closed | Author: changjonathanc (CONTRIBUTOR)

Fix #2449

@lhoestq Should I revert to the old `dummy/1.0.0` or  delete it and keep only `dummy/1.1.0`?

Comments: Hi ! Thanks for updating the version

> Should I revert to the old dummy/1.0.0 or delete it and keep only dummy/1.1.0?

Feel free to delete the old dummy data files
Keep original features order
State: closed | Author: albertvillanova (MEMBER)

When loading a Dataset from a JSON file whose column names are not sorted alphabetically, we should get the same column name order, whether we pass features (in the same order as in the file) or not.

I found this issue while working on #2366.

Comments: The arrow writer was supposing that the columns were always in the sorted order. I just pushed a fix to reorder the arrays accordingly to the schema. It was failing for many datasets like squad

and obviously it broke everything

Feel free to revert my commit. I can investigate this in the coming days

@lhoestq I do not understand when you say:
> It was failing for many datasets like squad

All the tests were green after my last commit.

> All the tests were green after my last commit.

Yes but loading the actual squad dataset was failing :/
BLUE file not found
State: closed | Author: mirfan899 (NONE)

Hi, I'm having the following issue when I try to load the `blue` metric.

```shell
import datasets
metric = datasets.load_metric('blue')
Traceback (most recent call last):
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 320, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.7.0/metrics/blue/blue.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 332, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/metrics/blue/blue.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 605, in load_metric
    dataset=False,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 343, in prepare_module
    combined_path, github_file_path
FileNotFoundError: Couldn't find file locally at blue/blue.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.7.0/metrics/blue/blue.py.
The file is also not present on the master branch on github.
```
Here is dataset installed version info
```shell
pip freeze | grep datasets
datasets==1.7.0
```


Comments: Hi ! The `blue` metric doesn't exist, but the `bleu` metric does.
You can get the full list of metrics [here](https://github.com/huggingface/datasets/tree/master/metrics) or by running
```python
from datasets import list_metrics

print(list_metrics())
```

Ah, my mistake. Thanks for correcting