noamaanMulla-03/datasets-issues huggingface dataset:
wikipedia.py generator that extracts XML doesn't release memory

I tried downloading Japanese wikipedia, but it always failed because of out of memory maybe.

I found that the generator function that extracts XML data in wikipedia.py doesn't release memory in the loop.

https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L464-L502

`root.clear()` intend to clear memory, but it doesn't.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L490
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L494
I replaced them with `elem.clear()`, then it seems to work correctly.

here is the notebook to reproduce it.
https://gist.github.com/miyamonz/dc06117302b6e85fa51cbf46dde6bb51#file-xtract_content-ipynb


Hi @miyamonz 
Thanks for investigating this issue, good job !
It would be awesome to integrate your fix in the library, could you open a pull request ?

OK! I'll send it later.
Optimize int precision

Optimize int precision to reduce dataset file size.

Close #1973, close #1825, close #861.


@lhoestq, are the tests OK? Some other cases I missed? Do you agree with this approach?

I just tested this and it works like a charm :) 

However tokenizing and then setting the format to "torch" to feed the tokens into a model doesn't seem to work anymore, since the pytorch tensors have the int32/int8 precisions instead of int64 that is required as model inputs.

For example:

```python
import torch
from datasets import Dataset
from transformers import BertModel, BertTokenizer

torch.set_grad_enabled(False)

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")

dataset = Dataset.from_dict({"text": ["hello there !"]})
dataset = dataset.map(tokenizer, input_columns="text", remove_columns=dataset.column_names)
dataset = dataset.with_format("torch")

print(dataset.features)
# {'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
#  'input_ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),  # this should be int32 though
#  'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None)}

model(**dataset[:1])
# RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.CharTensor instead (while checking arguments for embedding)

dataset = dataset.with_format("torch", dtype=torch.int64)

model(**dataset[:1])
# works as expected
```

Pinging @sgugger here to make sure we take the right decision here.

Do we want the "torch" format to always return int64 ? Or does it have to keep the precision defined by the `dataset.features` 
 and therefore we would need to specify "torch" with `dtype=torch.int64` ?

From a user perspective, I think it's fine if the "torch" format converts all ints types to `torch.int64` by default since it's what the model will need almost all the time. I don't see a case where you would want to keep the low precision at the top of my head, and one can always write a custom transform for an edge case.

Sounds good to me !
For consistency maybe we should make the float precision fixed as well (float32, I guess)

Yes, that would be the one used by default.

Do we have the same requirements for TensorFlow?

Yes I we should do the same for tensorflow as well since tf models would have the same issue

Thanks for adding this :)

@lhoestq I think this PR is ready... :)
Fix: Wikipedia - save memory by replacing root.clear with elem.clear

see: https://github.com/huggingface/datasets/issues/2031

What I did:
- replace root.clear with elem.clear
- remove lines to get root element
- $ make style
- $ make test
  - some tests required some pip packages, I installed them.

test results on origin/master and my branch are same. I think it's not related on my modification, isn't it?
```
==================================================================================== short test summary info ====================================================================================
FAILED tests/test_arrow_writer.py::TypedSequenceTest::test_catch_overflow - AssertionError: OverflowError not raised
============================================================= 1 failed, 2332 passed, 5138 skipped, 70 warnings in 91.75s (0:01:31) ==============================================================
make: *** [Makefile:19: test] Error 1

```

Is there anything else I should do?


The error you got is minor and appeared in the last version of pyarrow, we'll fix the CI to take this into account. You can ignore it
dataset(dgs): initial dataset loading script

When trying to create dummy data I get:

> Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has t o be created with less guidance. Make sure you create the file dummy_data.

I am not sure how to manually create the dummy_data (what exactly it should contain)

Also note, this library says:
> ImportError: To be able to use this dataset, you need to install the following dependencies['pympi'] using 'pip install pympi' for instance'

When you actually need to `pip install pympi-ling`



Hi @AmitMY, 

Were you able to figure this out?

I did not.
With all the limitations this repo currently has, I had to create a repo of my own using tfds to mitigate them. 
https://github.com/sign-language-processing/datasets/tree/master/sign_language_datasets/datasets/dgs_corpus

Closing as I don't know how to support this PR further
dataset(ngt): add ngt dataset initial loading script

Currently only making the paths to the annotation ELAN (eaf) file and videos available.
This is the first accessible way to download this dataset, which is not manual file-by-file.

Only downloading the necessary files, the annotation files are very small, 20MB for all of them, but the video files are large, 100GB in total, saved in `mpg` format. 
I do not intend to actually store these as an uncompressed array of frames, because it will be huge.

Future updates may add pose estimation files for all videos, making it easier to work with this data
dataset(wlasl): initial loading script

takes like 9-10 hours to download all of the videos for the dataset, but it does finish :) 


Followup: 
From the info in https://github.com/huggingface/datasets/pull/722, I probably should load the videos as array of frames directly into the database. 
This will make the dataset generation time very long, but will make working with the dataset much easier.

When I run:
```
python datasets-cli dummy_data datasets/wlasl
```

I get:
```
Checking datasets/wlasl/wlasl.py for additional imports. 
Found main folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl 
Found specific version folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786  
Found script file from datasets/wlasl/wlasl.py to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.py 
Found dataset infos file from datasets/wlasl/dataset_infos.json to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/dataset_infos.json 
Found metadata file for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.json 
Using custom data configuration default 
Loading Dataset Infos from /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786
Creating dummy folder structure for datasets/wlasl/dummy/0.3.0...   
Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data. 
Traceback (most recent call last):      
File "datasets-cli", line 36, in         
service.run()                                                                                                                                                                                                                                                                                                                                                                                                            File "/home/nlp/amit/anaconda2/envs/meta-scholar/lib/python3.7/site-packages/datasets-1.1.2-py3.7.egg/datasets/commands/dummy_data.py", line 73, in run 
for split in generator_splits: 
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```
Multilingual dIalogAct benchMark (miam)

My collaborators (@EmileChapuis, @PierreColombo) and I within the Affective Computing team at Telecom Paris would like to anonymously publish the miam dataset. It is assocated with a publication currently under review. We will update the dataset with full citations once the review period is over.


Hello. All aforementioned changes have been made. I've also re-run black on miam.py. :-)

I will run isort again. Hopefully it resolves the current check_code_quality test failure.

Once the review period is over, feel free to open a PR to add all the missing information ;)

Hi! I will follow up right now with one more pull request as I have new anonymous citation information to include.
Allow stateful function in dataset.map

Removes the "test type" section in Dataset.map which would modify the state of the stateful function. Now, the return type of the map function is inferred after processing the first example.

Fixes #1940 

@lhoestq Not very happy with the usage of `nonlocal`. Would like to hear your opinion on this.


@lhoestq Added a test. If you can come up with a better stateful callable, I'm all ears 😄.  

Sorry I said earlier that it was good to have it inside the loop, my mistake !

@lhoestq Okay, did some refactoring and now the "cache" part comes before the for loop. Thanks for the guidance.

Think this is ready for the final review.
load_from_disk takes a long time to load local dataset

I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)


Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?

It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])
    

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)
    
    return example
```


Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?

yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.

Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.

Yes this is a known issue: #625 
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`

Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?

After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.
MIAM dataset - new citation details

Hi @lhoestq, I have updated the citations to reference an OpenReview preprint.


Hi !
Looks like there's a unicode error in the new citation in the miam.py file.
Could you try to fix it ? Not sure from which character it comes from though

You can test if it works on your side with
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_miam
```

Unicode error resolved!
Fix deprecated warning message and docstring

Fix deprecated warnings:
- Use deprecated Sphinx directive in docstring
- Fix format of deprecated message
- Raise FutureWarning

Labels: name: documentation

I have a question: what about `dictionary_encode_column_`?
- It is deprecated in Dataset, but it recommends using a non-existing method instead: `Dataset.dictionary_encode_column` does not exist.
- It is NOT deprecated in DatasetDict.

`dictionary_encode_column_ ` should be deprecated since it never worked correctly. It will be removed in a major release.
This has to be deprecated in `DatasetDict` as well.
And `Dataset.dictionary_encode_column` doesn't exist indeed.

Thanks @lhoestq. I have fixed deprecated for `dictionary_encode_column_`.
Is there a way to use a GPU only when training an Index  in the process of add_faisis_index?

Motivation - Some FAISS indexes like IVF consist of the training step that clusters the dataset into a given number of indexes. It would be nice if we can use a GPU to do the training step and covert the index back to CPU as mention in [this faiss example](https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6).

Labels: name: question
Create README.md



@ManuelFay thanks you so much for adding a dataset card, this is such a cool contribution!

This looks like it uses an old template for the card we've moved things around a bit and we have an app you should be using to get the tags and the structure of the Data Fields paragraph :) Would you mind moving your text to the newer format (we're also asking contributors to keep the full template structure, even if some sections still have [More Information Needed] for the time being)

Here's the link to the instructions:
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Out of curiosity, what was your landing point for filling out the card? Did you follow the "Update on Github" when navigating the datasets? Trying to make the instructions as clear as possible :) 

@yjernite 

Perfect, I'll follow the instructions when I have a bit more time tomorrow ! I was actually browsing the new contributions after the dataset sprint and realized most of the "old" datasets were not tagged, so I just copied and pasted the readme from another dataset and was not aware there was precise instructions... Will fix !

BTW, amazing job with the retriBert work, I used the contrastive + in-batch negative quite a bit for various projects. Probably neither the time nor place to talk about that but I was curious as to why, in your original work, you prefered using a simple projection in the last layer to differentiate the question vs answer embedding, rather than allowing for bias in the dense layer or even just to fine-tune 2 different embedders for question + answer ? 

Cool! Looking forward to the next version!

Quick answer for retriBERT is that I expected a simple projection to generalize better and more importantly only having to store the gradients for the proj means training with larger batches :) If you want to keep chatting about it, feel free to send me an email!

Hi @ManuelFay ! 
If you're still interested in completing the FQuAD dataset card, note that we've generated one that is pre-filled.
Therefore feel free to complete it with the content you already have in your README.md.
This would be awesome ! And thanks again for your contribution :)

Yo @lhoestq , just not sure about the tag table at the top, I used @yjernite eli5 template so hope it's okay ! Also want to signal the streamlit app for dataset tagging has a weird behavior with the size categories when filling in the form. 

Thanks to you guys for doing that and sorry about the time it took, i completely forgot about it !
Fix: Allows a feature to be named "_type"

This PR tries to fix issue #1110. Sorry for taking so long to come back to this.

It's a simple fix, but i am not sure if it works for all possible types of `obj`. Let me know what you think @lhoestq 


Nice thank you !
This looks like a pretty simple yet effective fix ;)
Could you just add a test in `test_features.py` to make sure that you can create `features` with a `_type` field and that it is possible to convert it as a dict and reload it ?
```python
from datasets import Features, Value

# We usually use `asdict` on a `DatasetInfo` object which is a dataclass instance that contains the features.
# So we need the conversion of features to dict to work.
# You can test that using `dataclasses._asdict_inner`.
# This is the function used by `dataclasses.asdict` to convert a dataclass instance attribute to a dict
from dataclasses import _asdict_inner  

f = Features({"_type": Value("string")})
reloaded_f = Features.from_dict(_asdict_inner(f, dict))
assert reloaded_f == f
```

Sure, i will add a test. 
One question: are the posted benchmarks reliable? The extra type check seems to add quite some overhead judging by the relative differences. Do you think this is an issue?

The benchmark has a bit of noise, the values are fine ;)
especially in the change you did since the overhead added is negligible.

Ok, i added the test you described above. 

I avoided importing the private `_asdict_inner` method and directly used the `DatasetInfo` class, if this is ok with you. Thanks a lot for your support during this PR!
SQuAD version 

Hi~ 
I want train on squad dataset. What's the version of the squad? Is it 1.1 or 1.0? I'm new in QA, I don't find some descriptions about it. 


Hi ! This is 1.1 as specified by the download urls here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/datasets/squad/squad.py#L50-L55

Got it. Thank you~
Remove os.environ.copy in Dataset.map

Replace `os.environ.copy` with in-place modification
Fixes #2115 


I thought deepcopy on `os.environ` is unsafe (see [this](https://stackoverflow.com/questions/13142972/using-copy-deepcopy-on-os-environ-in-python-appears-broken)), but I can't replicate the behavior described in the linked SO thread.

Closing this one because #2119 has a much cleaner approach.
copy.deepcopy os.environ instead of copy

Fixes: https://github.com/huggingface/datasets/issues/2115

- bug fix: using envrion.copy() returns a dict.
- using deepcopy(environ) returns an `_environ` object
- Changing the datatype of the _environ object can break code, if subsequent libraries perform operations using apis exclusive to the environ object, like `environ.getenv()` for example.


Testing:

Tested the change on my terminal:

```
>>> import os
>>> x = deepcopy(os.environ)
>>> y = os.environ
>>> x is y
False
>>> isinstance(x, type(os.environ))
True
>>> z = os.environ.copy()
>>> isinstance(z, type(os.environ))
False
>>> isinstance(z, dict)
True
```
The datasets.map() implementation modifies the datatype of os.environ object

In our testing, we noticed that the datasets.map() implementation is modifying the datatype of python os.environ object from '_Environ' to 'dict'.

This causes following function calls to fail as follows:

`   
     x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    TypeError: get() takes no keyword arguments
`
It looks like the following line in datasets.map implementation introduced this functionality.

https://github.com/huggingface/datasets/blob/0cb1ac06acb0df44a1cf4128d03a01865faa2504/src/datasets/arrow_dataset.py#L1421

Here is the test script to reproduce this error. 


```
from datasets import load_dataset
from transformers import AutoTokenizer
import os


def test_train():
    model_checkpoint = "distilgpt2"
    datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    tokenizer.pad_token = tokenizer.eos_token


    def tokenize_function(examples):
        y = tokenizer(examples['text'], truncation=True, max_length=64)
        return y

    x = os.environ.get("TEST_ENV_VARIABLE_BEFORE_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_BEFORE_dataset_map {x}")
    print(f"Data type of os.environ before datasets.map = {os.environ.__class__.__name__}")
    datasets.map(tokenize_function, batched=True, num_proc=2, remove_columns=["text"])
    print(f"Data type of os.environ after datasets.map = {os.environ.__class__.__name__}")
    x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_AFTER_dataset_map {x}")


if __name__ == "__main__":
    test_train()


```
dataset viewer does not work anymore 

Hi
I normally use this link to see all datasets and how I can load them 


https://huggingface.co/datasets/viewer/

Now I am getting 

502 Bad Gateway
nginx/1.18.0 (Ubuntu)

could you bring this webpage back ? this was very helpful @lhoestq 
thanks for your help 

Labels: name: nlp-viewer

Thanks for reporting :) We're looking into it

Back up.