SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
I tried this on another machine (followed the same procedure I've mentioned above). This is what it shows (during the freeze period) for me:
```
>>> dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')
Downloading: 5.26kB [00:00, 1.23MB/s]                                                                                                                                    
Downloading: 1.40kB [00:00, 327kB/s]                                                                                                                                     
Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Connecting anonymously.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
```
After around 10 minutes, here's the loading of dataset:
```
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.42s/sources]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12sources/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14sources/s]
Dataset wiki40b downloaded and prepared to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f. Subsequent calls will reuse this data.
```
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Hi
I honestly also now tried on another machine and nothing shows up after
hours of waiting. Are you sure you have not set any specific setting? maybe
google cloud which seems it is used here, needs some credential setting?
thanks for any suggestions on this

On Tue, Mar 16, 2021 at 10:02 AM Bhavitvya Malik ***@***.***>
wrote:

> I tried this on another machine (followed the same procedure I've
> mentioned above). This is what it shows (during the freeze period) for me:
>
> >>> dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')
> Downloading: 5.26kB [00:00, 1.23MB/s]
> Downloading: 1.40kB [00:00, 327kB/s]
> Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
> WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
> Connecting anonymously.
> WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
>
> After around 10 minutes, here's the loading of dataset:
>
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.42s/sources]
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12sources/s]
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14sources/s]
> Dataset wiki40b downloaded and prepared to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f. Subsequent calls will reuse this data.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-800081772>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMX6A2ZTRZUIIZVFRCDTD4NC3ANCNFSM4ZA5R2UA>
> .
>
Issue: wikipedia.py generator that extracts XML doesn't release memory

Comment:
Hi @miyamonz 
Thanks for investigating this issue, good job !
It would be awesome to integrate your fix in the library, could you open a pull request ?
Issue: Loading a faiss index KeyError

Comment:
In your code `dataset2` doesn't contain the "embeddings" column, since it is created from the pandas DataFrame with columns "text" and "label".

Therefore when you call `dataset2[embeddings_name]`, you get a `KeyError`.

If you want the "embeddings" column back, you can create `dataset2` with
```python
dataset2 = load_from_disk(dataset_filename)
```
where `dataset_filename` is the place where you saved you dataset with the embeddings in the first place.
Issue: Loading a faiss index KeyError

Comment:
Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index 

I copy-pasted it here.

> When you are done with your queries you can save your index on disk:
> 
> ```python
> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
> ```
> Then reload it later:
> 
> ```python
> ds = load_dataset('crime_and_punish', split='train[:100]')
> ds.load_faiss_index('embeddings', 'my_index.faiss')
> ```
Issue: Loading a faiss index KeyError

Comment:
Hi !

The code of the example is valid.
An index is a search engine, it's not considered a column of a dataset.
When you do `ds.load_faiss_index("embeddings", 'my_index.faiss')`, it attaches an index named "embeddings" to the dataset but it doesn't re-add the "embeddings" column. You can list the indexes of a dataset by using `ds.list_indexes()`.

If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
This looks misleading indeed, and we should add a note to make it more explicit that it doesn't store the column that was used to build the index.

Feel free to open a PR to suggest an improvement on the documentation if you want to contribute :)
Issue: Loading a faiss index KeyError

Comment:
> If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
Yes. I was trying to use the dataset in RAG and it complained that the dataset didn't have the right columns. No problems when loading the dataset with `load_from_disk` and then doing `load_faiss_index`

What I learned was
1. column and index are different
2. loading the index does not create a column
3. the column is not needed to be able to use the index
4. RAG needs both the embeddings column and the index

If I can come up with a way to articulate this in the right spot in the docs, I'll open a PR
Issue: KeyError on using map after renaming a column

Comment:
Hi,

Actually, the error occurs due to these two lines:
```python
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')
```
`Dataset.rename_column` doesn't update the `_format_columns` attribute, previously defined by `Dataset.set_format`, with a new column name which is why this new column is missing in the output.
Issue: KeyError on using map after renaming a column

Comment:
Hi @mariosasko,

Thanks for opening a PR on this :)
Why does the old name also disappear?
Issue: KeyError on using map after renaming a column

Comment:
I just merged a @mariosasko 's PR that fixes this issue.
If it happens again, feel free to re-open :)
Issue: ValueError when rename_column on splitted dataset

Comment:
Hi,

This is a bug so thanks for reporting it. `Dataset.__setstate__`  is the problem, which is called when `Dataset.rename_column` tries to copy the dataset with `copy.deepcopy(self)`. This only happens if the `split` arg in `load_dataset` was defined as `ReadInstruction`.

To overcome this issue, use the named splits API (for now):
```python
train_ds, test_ds = load_dataset(
    path='csv',               
    delimiter='\t',          
    data_files=text_files,    
    split=['train[:90%]', 'train[-10%:]'],
)

train_ds = train_ds.rename_column('sentence', 'text')
```
Issue: ValueError when rename_column on splitted dataset

Comment:
This has been fixed in #2043 , thanks @mariosasko 
The fix is available on master and we'll do a new release soon :)

feel free to re-open if you still have issues
Issue: Interactively doing  save_to_disk and load_from_disk corrupts the datasets object?

Comment:
Hi,

Can you give us a minimal reproducible example? This [part](https://huggingface.co/docs/datasets/master/processing.html#controling-the-cache-behavior) of the docs explains how to control caching.
Issue: No upstream branch

Comment:
What's the issue exactly ?

Given an `upstream` remote repository with url `https://github.com/huggingface/datasets.git`, you can totally rebase from `upstream/master`.

It's mentioned at the beginning how to add the `upstream` remote repository

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L10-L14
Issue: No upstream branch

Comment:
~~What difference is there with the default `origin` remote that is set when you clone the repo?~~ I've just understood that this applies to **forks** of the repo 🤡
Issue: Local testing fails

Comment:
I'm not able to reproduce on my side.
Can you provide the full stacktrace please ?
What version of `python` and `dill` do you have ? Which OS are you using ?
Issue: Local testing fails

Comment:
```
co_filename = '<ipython-input-2-e0383a102aae>', returned_obj = [0]
                                                                                                                                                                       
    def create_ipython_func(co_filename, returned_obj):
        def func():
            return returned_obj
     
        code = func.__code__
>       code = CodeType(*[getattr(code, k) if k != "co_filename" else co_filename for k in code_args])
E       TypeError: an integer is required (got type bytes)

tests/test_caching.py:152: TypeError
```

Python 3.8.8 
dill==0.3.1.1
Issue: Local testing fails

Comment:
I managed to reproduce. This comes from the CodeType init signature that is different in python 3.8.8
I opened a PR to fix this test
Thanks !
Issue: Ambiguous documentation

Comment:
Hi @theo-m !

A few lines above this line, you'll find that the `_split_generators` method returns a list of `SplitGenerator`s objects:

```python
datasets.SplitGenerator(
    name=datasets.Split.VALIDATION,
    # These kwargs will be passed to _generate_examples
    gen_kwargs={
        "filepath": os.path.join(data_dir, "dev.jsonl"),
        "split": "dev",
    },
),
```

Notice the `gen_kwargs` argument passed to the constructor of `SplitGenerator`: this dict will be unpacked as keyword arguments to pass to the `_generat_examples` method (in this case the `filepath` and `split` arguments).

Let me know if that helps!
Issue: Ambiguous documentation

Comment:
Oh ok I hadn't made the connection between those two, will offer a tweak to the comment and the template then - thanks!
Issue: How to not load huggingface datasets into memory 

Comment:
So maybe a summary here: 
If I could fit a large model with batch_size = X into memory, is there a way I could train this model for huge datasets with keeping setting the same? thanks
Issue: How to not load huggingface datasets into memory 

Comment:
The `datastets` library doesn't load datasets into memory. Therefore you can load a dataset that is terabytes big without filling up your RAM.

The only thing that's loaded into memory during training is the batch used in the training step.
So as long as your model works with batch_size = X, then you can load an even bigger dataset and it will work as well with the same batch_size.

Note that you still have to take into account that some batches take more memory than others, depending on the texts lengths. If it works for a batch with batch_size = X and with texts of maximum length, then it will work for all batches.

In your case I guess that there are a few long sentences in the dataset. For those long sentences you get a memory error on your GPU because they're too long. By passing `max_train_samples` you may have taken a subset of the dataset that only contain short sentences. That's probably why in your case it worked only when you set `max_train_samples`.
I'd suggest you to reduce the batch size so that the batches with long sentences can be loaded on the GPU.

Let me know if that helps or if you have other questions
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
Adding to the previous information, I think `torch.utils.data.DataLoader` is doing some conversion. 
What I tried:
```python
train_dataset = load_dataset('mnist')
```
I don't use any `map` or `set_format` or any `transform`. I use this directly, and try to load batches using the `DataLoader` with batch size 2, I get an output like this for the `image`:

```
[[tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor...
```
For `label`, it works fine:
```
tensor([7, 6])
```
Note that I didn't specify conversion to torch tensors anywhere.

Basically, there are two problems here:
1. `dataset.map` doesn't return tensor type objects, even though it uses the transforms, the grayscale conversion in transform was done, but the output was lists only.
2.  The `DataLoader` performs its own conversion, which may be not desired.

I understand that we can't change `DataLoader` because it is a torch functionality, however, is there a way we can handle image data to allow using it with torch `DataLoader` and `torchvision` properly?

I think if the `image` was a torch tensor (N,H,W,C), or  a list of torch tensors (H,W,C), before it is passed to `DataLoader`, then we might not face this issue.
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
What's the feature types of your new dataset after `.map` ?

Can you try with adding `features=` in the `.map` call in order to set the "image" feature type to `Array2D` ?
The default feature type is lists of lists, we've not implemented shape verification to use ArrayXD instead of nested lists yet
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
Hi @lhoestq

Raw feature types are like this:
```
Image:
<class 'list'> 60000  #(type, len)
<class 'list'> 28
<class 'list'> 28
<class 'int'>
Label:
<class 'list'> 60000
<class 'int'>
```
Inside the `prepare_feature` method with batch size 100000 , after processing, they are like this:

Inside Prepare Train Features
```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After map, the feature type are like this:
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'list'> 28
<class 'float'>
Label:
<class 'list'> 60000
<class 'int'>
```

After dataloader with batch size 2, the batch features are like this:
```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```
<hr>

When I was setting the format of `train_dataset` to 'torch' after mapping  - 
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```

Corresponding DataLoader batch:
```
From DataLoader batch features
Image:
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

I will check with features and get back.
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
# Convert raw tensors to torch format
Strangely, converting to torch tensors works perfectly on `raw_dataset`:
```python
raw_dataset.set_format('torch',columns=['image','label'])
```
Types:
```
Image:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```

Using this for transforms:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                examples["image"][example_idx].numpy()
            ))
        else:
            images.append(examples["image"][example_idx].numpy())
        labels.append(examples["label"][example_idx])
    output = {"label":labels, "image":images}
    return output
```

Inside `prepare_train_features`:
```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After `map`:
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```
DataLoader batch:

```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

---

## Using `torch` format:
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```
DataLoader batches:

```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

---
## Using the features - `Array3D`:

```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After `map`:
```
Image:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```

After DataLoader `batch`:
```
Image:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

The last one works perfectly.

![image](https://user-images.githubusercontent.com/29076344/110491452-4cf09c00-8117-11eb-8a47-73bf3fc0c3dc.png)

I wonder why this worked, and others didn't.
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
Concluding, the way it works right now is:

1. Converting raw dataset to `torch` format.
2. Use the transform and apply using `map`, ensure the returned values are tensors. 
3. When mapping, use `features` with `image` being `Array3D` type.
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
What the dataset returns depends on the feature type.
For a feature type that is Sequence(Sequence(Sequence(Value("uint8")))), a dataset formatted as "torch" return lists of lists of tensors. This is because the lists lengths may vary.
For a feature type that is Array3D on the other hand it returns one tensor. This is because the size of the tensor is fixed and defined bu the Array3D type.
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
Okay, that makes sense.
Raw images are list of Array2D, hence we get a single tensor when `set_format` is used. But, why should I need to convert the raw images to `torch` format when `map` does this internally?

Using `Array3D` did not work with `map` when raw images weren't `set_format`ted to torch type.
Issue: Setting to torch format not working with torchvision and MNIST

Comment:
I understand that `map` needs to know what kind of output tensors are expected, and thus converting the raw dataset to `torch` format is necessary. Closing the issue since it is resolved.
Issue: Messages are being printed to the `stdout`

Comment:
This is expected to show this message to the user via stdout.
This way the users see it directly and can cancel the downloading if they want to.
Could you elaborate why it would be better to have it in stderr instead of stdout ?
Issue: Messages are being printed to the `stdout`

Comment:
@lhoestq, sorry for the late reply

I completely understand why you decided to output a message that is always shown. The only problem is that the message is printed to the `stdout`. For example, if the user runs `python run_glue.py > log_file`, it will redirect `stdout` to the file named  `log_file`, and the message will not be shown to the user.

Instead, we should print this message to `stderr`.  Even in the case of `python run_glue.py > log_file` only `stdout` is being redirected and so the message is always shown.
Issue: Empty evidence document ("provenance") in KILT ELI5 dataset

Comment:
Why did you close this issue? How did you end up finding the evidence documents? I'm running into a similar issue with other KILT tasks.
Issue: Windows Permission Error (most recent version of datasets)

Comment:
Hi @itsLuisa !

Could you give us more information about the error you're getting, please?
A copy-paste of the Traceback would be nice to get a better understanding of what is wrong :)
Issue: Windows Permission Error (most recent version of datasets)

Comment:
Hi @itsLuisa, thanks for sharing the Traceback.

You are defining the "id" field as a `string` feature:
```python
class Sample(datasets.GeneratorBasedBuilder):
    ...

    def _info(self):
        return datasets.DatasetInfo(
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    # ^^ here
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "pos_tags": datasets.Sequence(datasets.features.ClassLabel(names=[...])),
[...]
```

But in the `_generate_examples`, the "id" field is a list:
```python
ids = list()
```

Changing:
```python
"id": datasets.Value("string"),
```
Into:
```python
"id": datasets.Sequence(datasets.Value("string")),
```

Should fix your issue.

Let me know if this helps!
Issue: Error when exploring `arabic_speech_corpus`

Comment:
Actually soundfile is not a dependency of this dataset.
The error comes from a bug that was fixed in this commit: https://github.com/huggingface/datasets/pull/1767/commits/c304e63629f4453367de2fd42883a78768055532
Basically the library used to consider the `import soundfile` in the docstring as a dependency, while it's just here as a code example.

Updating the viewer to the latest version of `datasets` should fix this issue