helmo/github-issues huggingface dataset:
Creating custom dataset results in error while calling the map() function
State: closed | Author: GeetDsa (NONE)

calling `map()` of `datasets` library results into an error while defining a Custom dataset.
Reproducible example:
```
import datasets
class MyDataset(datasets.Dataset):

    def __init__(self, sentences):
        "Initialization"
        self.samples = sentences

    def __len__(self):
        "Denotes the total number of samples"
        return len(self.samples)

    def __getitem__(self, index):
        "Generates one sample of data"
        # Select sample
        # Load data and get label
        samples = self.samples[index]

        return samples

def preprocess_function_train(examples):
        inputs = examples
        labels = [example+tokenizer.eos_token for example in examples ]
        inputs = tokenizer(inputs, max_length=30, padding=True, truncation=True)
        labels = tokenizer(labels, max_length=30, padding=True, truncation=True)
        model_inputs = inputs
        model_inputs["labels"] = labels["input_ids"]
        print("about to return")
        return model_inputs


##train["sentence"] is dataframe column
train_dataset = MyDataset(train['sentence'].values.tolist())
train_dataset = train_dataset.map(
            preprocess_function,
            batched = True,
            batch_size=32
        )
```

Stack trace of error:
```
Traceback (most recent call last):
  File "dir/train_generate.py", line 362, in <module>
    main()
  File "dir/train_generate.py", line 245, in main
    train_dataset = train_dataset.map(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1244, in map
    return self._map_single(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 149, in wrapper
    unformatted_columns = set(self.column_names) - set(self._format_columns or [])
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 526, in column_names
    return self._data.column_names
AttributeError: 'MyDataset' object has no attribute '_data'
```

Comments: Hi,

the `_data` attribute is missing due to `MyDataset.__init__` not calling the parent `__init__`. However, I don't think it's a good idea to subclass the `datasets.Dataset` class (e.g. it's kind of dangerous to override `datasets.Dataset.__getitem__`). Instead, it's better to follow the "association over inheritance" approach with a simple wrapper class that delegates calls to a wrapped `Dataset` (map, etc.). Btw, the library offers the `datasets.Dataset.from_pandas` class method to directly create a `datasets.Dataset` from the dataframe.
The datasets.map() implementation modifies the datatype of os.environ object
State: closed | Author: leleamol (NONE)

In our testing, we noticed that the datasets.map() implementation is modifying the datatype of python os.environ object from '_Environ' to 'dict'.

This causes following function calls to fail as follows:

`   
     x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    TypeError: get() takes no keyword arguments
`
It looks like the following line in datasets.map implementation introduced this functionality.

https://github.com/huggingface/datasets/blob/0cb1ac06acb0df44a1cf4128d03a01865faa2504/src/datasets/arrow_dataset.py#L1421

Here is the test script to reproduce this error. 


```
from datasets import load_dataset
from transformers import AutoTokenizer
import os


def test_train():
    model_checkpoint = "distilgpt2"
    datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    tokenizer.pad_token = tokenizer.eos_token


    def tokenize_function(examples):
        y = tokenizer(examples['text'], truncation=True, max_length=64)
        return y

    x = os.environ.get("TEST_ENV_VARIABLE_BEFORE_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_BEFORE_dataset_map {x}")
    print(f"Data type of os.environ before datasets.map = {os.environ.__class__.__name__}")
    datasets.map(tokenize_function, batched=True, num_proc=2, remove_columns=["text"])
    print(f"Data type of os.environ after datasets.map = {os.environ.__class__.__name__}")
    x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_AFTER_dataset_map {x}")


if __name__ == "__main__":
    test_train()


```
Support for legal NLP datasets (EURLEX, ECtHR cases and EU-REG-IR)
State: closed | Author: iliaschalkidis (CONTRIBUTOR)

Add support for two legal NLP datasets:

- EURLEX (https://www.aclweb.org/anthology/P19-1636/)
- ECtHR cases (https://arxiv.org/abs/2103.13084)
- EU-REG-IR (https://arxiv.org/abs/2101.10726)

Comments: > Awesome thank you :)
> This is really cool
> 
> I left a few comments.
> 
> Also it looks like the dummy data are quite big (100-200KB each). Can you try to reduce their sizes please ? For example I noticed that all the jsonl files inside the `dummy_data.zip` files have 20 lines. Can you only keep 2 lines instead ?

Hi @lhoestq, I did my best to improve the README files, while I also decreased dummy data examples. I included one more legal dataset.

@lhoestq thanks for your review.

 I shortened the examples in README files and removed `DEFAULT_CONFIG_BUILDER` from `eu_regulatory_ir.py`.
Compute WER metric iteratively
State: closed | Author: albertvillanova (MEMBER)

Compute WER metric iteratively to avoid MemoryError.

Fix #2078.

Comments: I discussed with Patrick and I think we could have a nice addition: have a parameter `concatenate_texts` that, if `True`, uses the old implementation.

By default `concatenate_texts` would be `False`, so that sentences are evaluated independently, and to save resources (the WER computation has a quadratic complexity).

Some users might still want to use the old implementation.

@lhoestq @patrickvonplaten are you sure of the parameter name `concatenate_texts`? I was thinking about something like `iter`...

Not sure about the name, if you can improve it feel free to do so ^^'
The old implementation computes the WER on the concatenation of all the input texts, while the new one makes WER measures computation independent for each reference/prediction pair.
That's why I thought of `concatenate_texts`

@lhoestq yes, but the end user does not necessarily know the details of the implementation of the WER computation.

From the end user perspective I think it might make more sense: how do you want to compute the metric?
- all in once, more RAM memory needed?
- iteratively, less RAM requirements?

Because of that I was thinking of something like `iter` or `iterative`...

Personally like `concatenate_texts` better since I feel like `iter` or `iterate` are quite vague

Therefore, you can merge... ;)

Ok ! merging :)
Fix incorrect assertion in builder.py
State: closed | Author: dreamgonfly (CONTRIBUTOR)

Fix incorrect num_examples comparison assertion in builder.py

Comments: Hi ! The SplitInfo is not always available. By default you would get `split_info.num_examples == 0`
So unfortunately we can't use this assertion you suggested

> Hi ! The SplitInfo is not always available. By default you would get `split_info.num_examples == 0`
> So unfortunately we can't use this assertion you suggested

Then it would be better to just remove the assertion, because the existing assertion does nothing.
Add more issue templates and customize issue template chooser
State: closed | Author: albertvillanova (MEMBER)

When opening an issue, it is not evident for the users how to choose a blank issue template. There is a link at the bottom of all the other issue templates (`Don’t see your issue here? Open a blank issue.`), but this is not very visible for users. This is the reason why many users finally chose the `add-dataset` template instead (this is more visible) for issues that indeed are not requesting the addition of a new dataset.

~~With this PR, the default blank issue template would be as visible as the other templates (as the `add-dataset` template), thus making easier for the users to choose it.~~

With this PR:
- more issue templates, besides `add-dataset`, are added: `bug-report` and `feature-request`
- the issue template chooser is customized, so that it now includes a link to `Discussions` for questions

Comments: If you agree, I could also add a link to [Discussions](https://github.com/huggingface/datasets/discussions) in order to reinforce the use of Discussion to make Questions (instead of Issues).

I could also add some other templates: Bug, Feature Request,...

@theo-m we wrote our same comments at the same time... 😉
Is there a way to use a GPU only when training an Index  in the process of add_faisis_index?
State: open | Author: shamanez (NONE)
Labels: name: question

Motivation - Some FAISS indexes like IVF consist of the training step that clusters the dataset into a given number of indexes. It would be nice if we can use a GPU to do the training step and covert the index back to CPU as mention in [this faiss example](https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6).
Metadata validation
State: closed | Author: theo-m (CONTRIBUTOR)

- `pydantic` metadata schema with dedicated validators against our taxonomy
- ci script to validate new changes against this schema and start a vertuous loop
- soft validation on tasks ids since we expect the taxonomy to undergo some changes in the near future

for reference with the current validation we have ~365~ 378 datasets with invalid metadata! full error report [_here_.](https://gist.github.com/theo-m/61b3c0c47fc6121d08d3174bd4c2a26b)

Comments: > Also I was wondering this is really needed to have `utils.metadata` as a submodule of `datasets` ? This is only used by the CI so I'm not sure we should have this in the actual `datasets` package.

I'm unclear on the suggestion, would you rather have a root-level `./metadata.py` file? I think it's well where it is, if anything we could move it out of utils and into `datasets` as it could be used by e.g. `DatasetDict` so that users can pull the metadata easily rather than have to reparse the readme.


Ok that makes sense if we want to have functions that parse the metadata for users

Hi @theo-m @lhoestq 

This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)

Sorry for the delay in responding.

Thanks,
Gunjan

> Hi @theo-m @lhoestq
> 
> This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)
> 
> Sorry for the delay in responding.
> 
> Thanks,
> Gunjan

Hi @gchhablani, yes I think at the moment the best solution is for you to write in `datasets-tagging`, as the PR will allow us to discuss and review, even though the work will be ported to this repo in the end. 
Or we wait for this to be merged and you reopen the PR here, your call :)

cc @abhi1thakur
Request to remove S2ORC dataset
State: open | Author: kyleclo (NONE)

Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!

Comments: Hello @kyleclo! Currently, we are getting the data from your bucket, so if you remove it the HF script won't work anymore :) 

Until you solve things on your end, @lhoestq suggested we just return a warning message when people try to load that dataset from HF. What would you like it to say?

Hi @kyleclo, as of today, you have not removed your bucket data yet, and therefore HuggingFace can download it from there.

Is it OK? Are you planning to eventually delete it? Thank you.

Hi!  Sorry I missed @yjernite 's previous message, thanks for responding!  

Is there an option where we can keep our data in our bucket, but the HF script no longer pulls data from it?
Trouble loading wiki_movies
State: closed | Author: adityaarunsinghal (NONE)

Hello,
I am trying to load_dataset("wiki_movies") and it gives me this error - 

`FileNotFoundError: Couldn't find file locally at wiki_movies/wiki_movies.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/wiki_movies/wiki_movies.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/wiki_movies/wiki_movies.py`

Trying to do `python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wiki_movies \` also gives the same error. 

Is this something on my end? From what I can tell, this dataset was re-added by @lhoestq a few months ago. 
Thank you!

Comments: Hi ! `wiki_movies` was added in `datasets==1.2.0`. However it looks like you have `datasets==1.1.2`.

To use `wiki_movies`, please update `datasets` with
```
pip install --upgrade datasets
```

Thanks a lot! That solved it and I was able to upload a model trained on it as well :)
citation, homepage, and license fields of `dataset_info.json` are duplicated many times
State: closed | Author: samsontmr (NONE)
Labels: name: enhancement

name: good first issue

This happens after a `map` operation when `num_proc` is set to `>1`. I tested this by cleaning up the json before running the `map` op on the dataset so it's unlikely it's coming from an earlier concatenation.

Example result:
```
"citation": "@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n
```

@lhoestq and I believe this is happening due to the fields being concatenated `num_proc` times.

Comments: Thanks for reporting :)
Maybe we can concatenate fields only if they are different.

Currently this is done here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/info.py#L180-L196

This can be a good first contribution to the library.
Please comment if you'd like to improve this and open a PR :)
MIAM dataset - new citation details
State: closed | Author: eusip (CONTRIBUTOR)

Hi @lhoestq, I have updated the citations to reference an OpenReview preprint.

Comments: Hi !
Looks like there's a unicode error in the new citation in the miam.py file.
Could you try to fix it ? Not sure from which character it comes from though

You can test if it works on your side with
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_miam
```

Unicode error resolved!
Fix deprecated warning message and docstring
State: closed | Author: albertvillanova (MEMBER)
Labels: name: documentation

Fix deprecated warnings:
- Use deprecated Sphinx directive in docstring
- Fix format of deprecated message
- Raise FutureWarning

Comments: I have a question: what about `dictionary_encode_column_`?
- It is deprecated in Dataset, but it recommends using a non-existing method instead: `Dataset.dictionary_encode_column` does not exist.
- It is NOT deprecated in DatasetDict.

`dictionary_encode_column_ ` should be deprecated since it never worked correctly. It will be removed in a major release.
This has to be deprecated in `DatasetDict` as well.
And `Dataset.dictionary_encode_column` doesn't exist indeed.

Thanks @lhoestq. I have fixed deprecated for `dictionary_encode_column_`.
load_from_disk takes a long time to load local dataset
State: closed | Author: samsontmr (NONE)

I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)

Comments: Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?

It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])
    

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)
    
    return example
```


Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?

yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.

Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.

Yes this is a known issue: #625 
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`

Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?

After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.
SQuAD version 
State: closed | Author: h-peng17 (NONE)

Hi~ 
I want train on squad dataset. What's the version of the squad? Is it 1.1 or 1.0? I'm new in QA, I don't find some descriptions about it. 

Comments: Hi ! This is 1.1 as specified by the download urls here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/datasets/squad/squad.py#L50-L55

Got it. Thank you~
CoNLL 2003 dataset not including German
State: closed | Author: rxian (NONE)
Labels: name: dataset request

Hello, thanks for all the work on developing and maintaining this amazing platform, which I am enjoying working with!

I was wondering if there is a reason why the German CoNLL 2003 dataset is not included in the [repository](https://github.com/huggingface/datasets/tree/master/datasets/conll2003), since a copy of it could be found in some places on the internet such as GitHub? I could help adding the German data to the hub, unless there are some copyright issues that I am unaware of...

This is considering that many work use the union of CoNLL 2002 and 2003 datasets for comparing cross-lingual NER transfer performance in `en`, `de`, `es`, and `nl`. E.g., [XLM-R](https://www.aclweb.org/anthology/2020.acl-main.747.pdf).

## Adding a Dataset
- **Name:** CoNLL 2003 German
- **Paper:** https://www.aclweb.org/anthology/W03-0419/
- **Data:** https://github.com/huggingface/datasets/tree/master/datasets/conll2003


Comments: Hello. I've been looking for information about German Conll2003 and found your question. Official site (https://www.clips.uantwerpen.be/conll2003/ner/) mentions that organizers provide only annotation. German texts (ECI Multilingual Text Corpus) are not freely available and can be ordered from the Linguistic Data Consortium.

But maybe something has changed since 2003.

You can find the reason for not including the German data here: https://github.com/huggingface/datasets/issues/4230.
Fix: Allows a feature to be named "_type"
State: closed | Author: dcfidalgo (CONTRIBUTOR)

This PR tries to fix issue #1110. Sorry for taking so long to come back to this.

It's a simple fix, but i am not sure if it works for all possible types of `obj`. Let me know what you think @lhoestq 

Comments: Nice thank you !
This looks like a pretty simple yet effective fix ;)
Could you just add a test in `test_features.py` to make sure that you can create `features` with a `_type` field and that it is possible to convert it as a dict and reload it ?
```python
from datasets import Features, Value

# We usually use `asdict` on a `DatasetInfo` object which is a dataclass instance that contains the features.
# So we need the conversion of features to dict to work.
# You can test that using `dataclasses._asdict_inner`.
# This is the function used by `dataclasses.asdict` to convert a dataclass instance attribute to a dict
from dataclasses import _asdict_inner  

f = Features({"_type": Value("string")})
reloaded_f = Features.from_dict(_asdict_inner(f, dict))
assert reloaded_f == f
```

Sure, i will add a test. 
One question: are the posted benchmarks reliable? The extra type check seems to add quite some overhead judging by the relative differences. Do you think this is an issue?

The benchmark has a bit of noise, the values are fine ;)
especially in the change you did since the overhead added is negligible.

Ok, i added the test you described above. 

I avoided importing the private `_asdict_inner` method and directly used the `DatasetInfo` class, if this is ok with you. Thanks a lot for your support during this PR!