SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
@lhoestq +1 for the `Dataset.from_generator` idea.

Having thought about it, let's avoid adding `Dataset.from_iterable` to the API since dictionaries are technically iteralbles ("iterable" is a broad term in Python), and we already provide `Dataset.from_dict`. And for lists maybe we can add `Dataset.from_list` similar to `pa.Table.from_pylist`. WDYT?
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
Hi @StephennFernandes!

To fix the issues in the copied code, rename `generate_examples` to` _generate_examples` and add one level of indentation as this is a method of `GeneratorBasedBuilder` and define `_split_generators` as follows (again as a method of `GeneratorBasedBuilder):
```python
 def _split_generators(self, dl_manager):
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                gen_kwargs={},
            ),
        ]
```

And if you are feeling extra adventurous, you can try to use ArrowWriter to directly create a cache file:
```python
from datasets import Dataset
from datasets.arrow_writer import ArrowWriter

writer = ArrowWriter(path="path/to/cache_file.arrow", writer_batch_size=1000)

with writer:
    for ex in generator:
        writer.write(ex) 
    writer.finalize()

dset = Dataset.from_file("path/to/cache_file.arrow")
```
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
I have a problem which I think is very similar: I would like to "stream" data to a HF Array (memory-mapped) Dataset, where the final size of the dataset is unknown, but could be much larger than what fits into memory.
What I want to end up with is an Array Dataset which I can open using `Dataset.load_from_disk(dataset_path="somename")` and use e.g. as the training set. 

For this I would have thought there should be an API which allows me to open/create the dataset (and define the features etc), then write examples to the dataset, but I could not find a way to do this. 

I tried doing this and it looks like it works, but it feels very hacky and I am not sure if this might fail to update some of the fields in the json files which may turn out to be important:
```
from datasets import Dataset, Features, ClassLabel, Sequence, Value
from datasets.arrow_writer import ArrowWriter                                                                                                                                                   
# 1) define the features
features = Features(dict(
    id=Value(dtype="string"),
    tokens=Sequence(feature=Value(dtype="string")),
    ner_tags=Sequence(feature=ClassLabel(names=['O', 'B-corporation', 'I-corporation', 'B-creative-work', 'I-creative-work', 'B-group', 'I-group', 'B-location', 'I-location', 'B-person', 'I-person', 'B-product', 'I-product'])),
))
# 2) create empty dataset for examples with these features and store to disk
empty = dict(
    id = [],
    tokens = [],
    ner_tags = [],
)
ds = Dataset.from_dict(empty, features=features)
ds.save_to_disk(dataset_path="debug_ds1")

# 3) directly write all the examples to the arrow dataset 
with ArrowWriter(path="debug_ds1/dataset.arrow") as writer: 
    writer.write(dict(id=0, tokens=["a", "b"], ner_tags=[0, 0])) 
    writer.write(dict(id=1, tokens=["x", "y"], ner_tags=[1, 0])) 
    writer.finalize() 
 
ds2 = Dataset.load_from_disk(dataset_path="debug_ds1")
len(ds2)
```
Is there a cleaner/proper way to do this?

I like the sound of `Dataset.from_iterable` or `Dataset.from_generator` (should not from iterable be able to handle from generator too as all generators are iterables?) but how would I define the features for me examples there?
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
Hi @johann-petrak! You can pass the features directly to ArrowWriter's initializer like so `ArrowWriter(..., features=features)`.

And the reason why I prefer `Dataset.from_generator` over `Dataset.from_iterable` is mentioned in one of my previous comments.
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
@mariosasko so at the moment we still have to create a fake `Dataset` first and then use `ArrowWriter` to write an actual dataset? I'm using the latest version of `datasets` on pypi but my final file is always empty. Is there anything wrong with the code below?

```python
    total = 0
    with ArrowWriter(path=str(final_data_path), features=features) as writer:
        for batch in loader:
            for traj in batch:
                for generator in question_generators:
                    for xi in generator(traj):
                        # print(f"Question: {xi.question}, answer: {xi.answer}")
                        total += 1
                        writer.write(
                            {
                                "id": f"qa_{total}",
                                "question": xi.question,
                                "answer": xi.answer,
                            }
                        )
        writer.finalize()
    print(f"Total #questions = {total}") # this prints 402
```
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
This works for me if I then (actually I also close the writer: `writer.close()`)  open the Arrow file as a dataset using `ds=Dataset.from_file(final_data_path)` then `ds.save_to_disk(somedir)`. The Dataset created that way contains the expected examples.
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
Oh thanks. That did the trick I believe. Shouldn't ArrowWriter have a context manager that does these operations?
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
You can just use `Dataset.from_file` to get your dataset, no need to do an extra `save_to_disk` somewhere else ;)
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
I was thinking that `save_to_disk` is necessary when one wants to re-use that dataset as a proper HF dataset later, no?
At least what I wanted to achieve is create a dataset that can be opened like any other local or remote dataset.
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
`save_to_disk`/`load_from_disk` is indeed more general, e.g. it supports datasets that consist in several files, and saves some extra info in a dataset_info.json file (description, citation, split sizes, etc.)

If you have one single file it's fine to simply do `.from_file()`
Issue: Dataset Viewer issue for ett

Comment:
Thanks for reporting @dgcnz.

I have checked that the dataset works fine in streaming mode.

Additionally, other datasets containing timestamps are properly rendered by the viewer: https://huggingface.co/datasets/blbooks

I have tried to force the refresh of the preview, but the endpoint is not responsive: Connection timed out

CC: @severo
Issue: Dataset Viewer issue for ett

Comment:
I've just resent the refresh of the preview to the new endpoint, without success.

CC: @severo
Issue: Dataset Viewer issue for conll2012_ontonotesv5

Comment:
Thanks for reporting, @jiangwy99.

I guess this could be addressed only once we fix our issue with irresponsive backend endpoint.

CC: @severo
Issue: Dataset Viewer issue for conll2012_ontonotesv5

Comment:
Fixed, thanks for the patience. The issue was the amount of RAM allowed to extract the first rows of the dataset was not sufficient.
Issue: [TypeError: Couldn't cast array of type] Cannot process dataset in v2.2.2

Comment:
And if the problem is that the way I am to construct the {Entity Type: list of spans} makes entity types without any spans hard to handle, is there a better way to meet the demand? Although I have verified that to make entity types without any spans to behave like `entity_chunk[label] = [[""]]` can perform normally, I still wonder if there is a more elegant way?
Issue: Dataset should have a `.name` field

Comment:
Hi! You can already use `dset.builder_name` and `dset.config_name` for that purpose. And when it comes to versioning, it's better to use `dset._fingerprint` than the `version` attribute as the former represents a deterministic hash that encodes all the mutable ops executed on a dataset, and the latter stays the same unless it's manually updated after each op.
Issue: Dataset should have a `.name` field

Comment:
@mariosasko  Can we make ._fingerprint not private? seems a critical component for tracking how a model was generated to ensure reproducibility.
Issue: "NonMatchingChecksumError" when importing 'spider' dataset

Comment:
Thanks for reporting, @OmarAlaaeldein.

Datasets hosted at Google Drive give problems quite often due to a change in their service:
- #3786 

Related to:
- #3906

I'm having a look.
Issue: "NonMatchingChecksumError" when importing 'spider' dataset

Comment:
We have made a Pull Request to replace the Google Drive URL. This fix will be accessible in our next `datasets` library release.

In the meantime, once the PR merged into master, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Issue: load dataset wikitext-2-raw-v1 failed. Could not reach wikitext-2-raw-v1.py.

Comment:
I tried in this way.

```python
from datasets import load_dataset
dataset = load_dataset(path="wikitext", name="wikitext-103-v1", split="train")
```
Issue: LocalDatasetModuleFactoryWithoutScript extracts invalid builder name

Comment:
Ok, so
```
os.path.basename("/home/user/")
```
gives `''` while 
```
os.path.basename("/home/user")
```
gives `user`. 
The code should check if the last char is a slash.
Issue: LocalDatasetModuleFactoryWithoutScript extracts invalid builder name

Comment:
I came through the same issue , just removing the last slash in the dataset path fixed it for me, may be this repo moderators could accept this as an accepted answer atleast if this could not be integrated

> The fix is:
> 
> ```
> "name": os.path.basename(self.path[:-1] if self.path[-1] == "/" else self.path)
> ```

@apohllo consider making a pull request on this 

Thanks for the amazing contributions from huggingface people !!
Issue: LocalDatasetModuleFactoryWithoutScript extracts invalid builder name

Comment:
@mariosasko here we go:

https://github.com/huggingface/datasets/pull/4967

TBH I haven't tested it yet, but should work, since this is a basic change.
Issue: Calling `cast_column`/`remove_columns` and a sequence of `map` operations ends up making `faiss` fail with `ValueError`

Comment:
It works if we either remove the `ds = ds.cast_column("id", Value("int32"))` line from the code above, or if instead calling `ds.remove_columns()` we remove the columns inside each mapping as `ds.map(..., remove_columns=[...])` instead of right after the mapping.

Both of those solutions seem to fix the issue, so the root cause of it may be around that. Sorry I cannot provide you more insights, in case I get to fix it I'll submit a PR, in the meanwhile the code that I'm using as a workaround is the following:

```python
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32"))
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings, remove_columns=["inputs"])
ds.add_faiss_index(column="embeddings")
```
Issue: Calling `cast_column`/`remove_columns` and a sequence of `map` operations ends up making `faiss` fail with `ValueError`

Comment:
FYI the main reason I want to use `dataset.remove_columns` rather than the function inside `dataset.map` is because according to the 🤗 Datasets documentation, it's faster.

"🤗 Datasets also has a [Dataset.remove_columns()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset.remove_columns) method that is functionally identical, but faster, because it doesn’t copy the data of the remaining columns."

More information at https://huggingface.co/docs/datasets/process#map
Issue: Calling `cast_column`/`remove_columns` and a sequence of `map` operations ends up making `faiss` fail with `ValueError`

Comment:
So on, I've created #4411 so as to fix the bug with `remove_columns` under certain conditions before `add_faiss_index`, which means that the scenarios not working above are already working fine.
Issue: trainer became extremely slow after reload dataset by `load_from_disk`

Comment:
I tried to make the dataset much more smaller (100000 rows) , then the speed became `33.88it/s` from`8.62s/it`.  It's nearly 200 times... Do you have any idea? Thank you!
Issue: trainer became extremely slow after reload dataset by `load_from_disk`

Comment:
Similar issue: https://github.com/huggingface/transformers/issues/8818

I changed `RandomSampler` to  `SequentialSampler` in the `trainer.py`, but the speed didn't become faster.
Issue: trainer became extremely slow after reload dataset by `load_from_disk`

Comment:
I changed
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
```
to
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset", keep_in_memory=True
    )
```
and obtained normal speed. It's seems that the problem is on the os's speed limit.
Issue: trainer became extremely slow after reload dataset by `load_from_disk`

Comment:
Hi ! Currently `save_to_disk` saves one big Arrow file, which causes some slow downs. This has been discussed in #3735 and we'll implement sharding pretty soon to solve this

For now you can try splitting and saving your dataset in several Arrow files. Then you can load them one by one and use `concatenate_datasets` to have your big dataset again and hopefully with a better speed
Issue: Bug for wiki_auto_asset_turk from GEM

Comment:
Hi @StevenTang1998,

We have fixed the issue:
- #4389

The fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by installing `datasets` from our GitHub repo:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Issue: Bug for wiki_auto_asset_turk from GEM

Comment:
Thanks for your reply!!
And the totto dataset has the same problem. The url should be change to [https://storage.googleapis.com/totto-public/totto_data.zip](https://storage.googleapis.com/totto-public/totto_data.zip).
Issue: Bug for wiki_auto_asset_turk from GEM

Comment:
Hi again @StevenTang1998,

I don't see any problem when loading `totto` dataset:
```python
In [4]: import datasets
   ...: ds = datasets.load_dataset("totto")
Downloading builder script: 5.58kB [00:00, 5.33MB/s]                                                                                                                                                       
Downloading metadata: 2.78kB [00:00, 2.96MB/s]                                                                                                                                                             
Using custom data configuration default
Downloading and preparing dataset totto/default (download: 179.03 MiB, generated: 706.59 MiB, post-processed: Unknown size, total: 885.62 MiB) to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188M/188M [00:32<00:00, 5.77MB/s]
Dataset totto downloaded and prepared to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 147.95it/s]

In [5]: ds
Out[5]: 
DatasetDict({
    train: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 120761
    })
    validation: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
    test: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
})
```
Issue: Bug in caching 2 datasets both with the same builder class name

Comment:
Hi @NouamaneTazi, thanks for reporting.

Please note that both datasets are cached in the same directory because their loading builder classes have the same name: `class MTOP(datasets.GeneratorBasedBuilder)`.

You should name their builder classes differently, e.g.:
- `MtopDomain`
- `MtopIntent`
Issue: Bug in caching 2 datasets both with the same builder class name

Comment:
Hi @NouamaneTazi, please note that after our fix:
- #4388

we do not consider the class name anymore, but the name of the file where the loading builder class is implemented.
Issue: Latest dill release raises exception

Comment:
Just an additional insight, the latest dill (either 0.3.5 or 0.3.5.1) also broke the hashing/fingerprinting of any mapping function.

For example:
```
from datasets import load_dataset

d = load_dataset("rotten_tomatoes")
d.map(lambda x: x)
```

Returns the standard non-dillable error:
```
Parameter 'function'=<function <lambda> at 0x7fe7d18c9560> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly....
```
Issue: Latest dill release raises exception

Comment:
Thanks a lot @gugarosa for the insight: we will incorporate it in our CI as regression testing for future dill releases.
Issue: Latest dill release raises exception

Comment:
@albertvillanova

I did a deep dive into @gugarosa's problem and found the issue and it might be related to the one @sgugger discovered. In dill 0.3.5(.1), I created a new `save_function` that fixes a bug in dill that prevented the pickling of recursive inner functions. It was a more complete solution to the problem that `dill._dill.stack` tried to solve in the internal API of dill. Since `dill._dill.stack` was no longer needed, I removed it. Since datasets copies the `save_function` directly from the dill API, it stops working with the new dill version since `dill._dill.stack` is no longer present and the `save_function` has been updated with new code.

https://github.com/huggingface/datasets/blob/95193ae61e92aa537d0c65d37a1fd9d2393aae89/src/datasets/utils/py_utils.py#L607-L678

~If the dill version is below 0.3.5, you should keep this function. If it is after, you would need to update your copy of `save_function` to use the code I introduced, or manually add a `stack` variable to `dill._dill` if it doesn't exist. Fortunately, in any version of Python 3.7+, dictionaries are always in insertion order and dill no longer supports Python 3.6 or older. So, any globals dictionary saved by dill 0.3.5+ will be deterministic given that the version of dill is held constant and this save_function is unnecessary for newer versions of dill.~

Ah. I see what is happening. I guess a different copy of the function code is needed that sorts the global variables by name.

```py
if dill.__version__.split('.') < ['0', '3', '5']:
    # current save_function code inside here
else:
    # new save_function code inside here with the following line inserted after creating the globals
    globs = {k: globs[k] for k in sorted(globs.keys())} 
```

Will look into the test case @sgugger pointed out after that and verify if this is causing the problem.

I am actually looking into rewriting the global variables code in uqfoundation/dill#466 and will keep this in mind and will try to create an easy way to modify the global variables in dill 0.3.6 (for example, sort them by key like datasets does).
Issue: Latest dill release raises exception

Comment:
Thanks a lot for your investigation @anivegesana.

Yes, we copied-pasted the old `save_function` function from `dill`, just adding a line to make deterministic the order of global variables `globs`. 

However, this function has changed a lot from version 0.3.5, after your PR (thank you for the fix in recursiveness, indeed):
- uqfoundation/dill#443

We have to address this change.

If finally your PR to sort global variables is merged into dill 0.3.6, that will make our life easier, as the tweak will no longer be necessary. ;)

I have included a regression test so that we are sure future releases of dill do not break `datasets`:
- #4385
Issue: Latest dill release raises exception

Comment:
I should note that because Python 3.6 and older are now deprecated and Python 3.7 has insertion order dictionaries, the globals in dill will have a deterministic order, just not sorted. I would still keep it sorted like you have it to help with stability (for example, if someone reorders variables in a file, then sorting the globals would not invalidate the cache.)

It seems that the order is not quite deterministic in IPython. Huggingface datasets seems to do well in Jupyter regardless, so it is not a good idea to remove the sorting. uqfoundation/dill#19