SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Add URL field to Wikimedia dataset instances: wikipedia,...

Comment:
Thanks @geohci.

I had already been looking for information about the conversion from title to URL and I found that apart from replacing blanks with underscores, some other special character must also be percent-encoded (e.g. `"` to `%22`): https://meta.wikimedia.org/wiki/Help:URL

Therefore, I have finally used `urllib.parse.quote` function. This additionally percent-encodes non-ASCII characters, but Wikimedia docs say these are equivalent:
> For the other characters either the code or the character can be used in internal and external links, they are equivalent. The system does a conversion when needed.
> [[%C3%80_propos_de_M%C3%A9ta]]
> is rendered as [À_propos_de_Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), almost like [À propos de Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), which leads to this page on Meta with in the address bar the URL
> [http://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta)
> while [http://meta.wikipedia.org/wiki/À_propos_de_Méta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta) leads to the same.
Issue: Install Audio dependencies to support audio decoding

Comment:
https://huggingface.co/datasets/projecte-aina/parlament_parla -> works (but we still have to show an audio player)

https://huggingface.co/datasets/openslr -> another issue: `Message:       [Errno 2] No such file or directory: '/home/hf/datasets-preview-backend/zip:/asr_javanese/data/00/00004fe6aa.flac'`
Issue: Install Audio dependencies to support audio decoding

Comment:
But https://huggingface.co/datasets/openslr/viewer does not work

<img width="678" alt="Capture d’écran 2022-04-12 à 13 59 46" src="https://user-images.githubusercontent.com/1676121/162958013-e31ef2ae-f886-47b7-9f27-664ed3d4b5a1.png">

Same issue as #4126:

```
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'
```
Issue: Preserve all feature types when saving a dataset on the Hub with `push_to_hub`

Comment:
According to this [comment in the forum](https://discuss.huggingface.co/t/save-datasetdict-to-huggingface-hub/12075/8?u=lhoestq), using `push_to_hub` on a dataset with `ClassLabel` can also make the feature simply disappear when it's reloaded !
Issue: Preserve all feature types when saving a dataset on the Hub with `push_to_hub`

Comment:
Maybe we can also fix https://github.com/huggingface/datasets/issues/3035 while working on this because, as pointed out in my initial post, `save_to_disk` also saves the `dataset_info.json`  file.
Issue: Dataset viewer issue for `dansbecker/hackernews_hiring_posts`

Comment:
This issue was fixed by me calling `all_datasets.push_to_hub("hackernews_hiring_posts")`.

The previous problems were from calling `all_datasets.save_to_disk` and then pushing with `my_repo.git_add` and `my_repo.push_to_hub`.
Issue: Add EDGAR

Comment:
Datasets are not tracked in this repository anymore. But you can make your own dataset in the huggingface hub
Issue: None batched `with_transform`, `set_transform`

Comment:
Hi ! Thanks for the suggestion :)
It makes sense to me, and it can surely be implemented by wrapping the user's function to make it a batched function. However I'm not a big fan of the inconsistency it would create with `map`: `with_transform` is batched by default while `map` isn't.

Is there something you would like to contribute ? I can give you some pointers if you want
Issue: None batched `with_transform`, `set_transform`

Comment:
Hi @lhoestq ,
Sorry I missed your reply.

I would love to contribute. But I don't know which solution would be the best for this repo.

> However I'm not a big fan of the inconsistency it would create with map: with_transform is batched by default while map isn't.

I agree. What do you think about the alternative solutions?

> * Convert a non-batched transform function to batched one myself.

This won't be able to use torch loader multi-worker.

> * Wrap a 🤗 Dataset with torch Dataset, and add a __getitem__. 🙄

This is actually pretty simple.

```python
import torch

class LazyMapTorchDataset(torch.utils.data.Dataset):
    def __init__(self, ds, fn):
        self.ds = ds
        self.fn = fn
    def __getitem__(self, i):
        return self.fn(self.ds[i])

d = [{1:2, 2:3}, {1:3, 2:4}]
ds = LazyMapTorchDataset(d, lambda x:{k:v*2 for k,v in x.items()})
for i in range(2):
    print(f'before {d[i]}')
    print(f'after {ds[i]}')
```
```
before {1: 2, 2: 3}
after {1: 4, 2: 6}
before {1: 3, 2: 4}
after {1: 6, 2: 8}
```

But this requires converting data to torch tensor myself. And this is really similar to `.map()`, why not just use it? So I have the next solution.

> * Have lazy=False in Dataset.map, and returns a LazyDataset if lazy=True. This way the same map interface can be used, and existing code can be updated with one argument change.

I think I like this solution best.  Because `.with_transform` is entangled with `.with_format`, so seems more flexible to modify the `.map` than to modify `.with_transform`.

The usage looks nice, too.
```python
# lazy, one to one, can be parallelized via torch loader, no need to set `num_worker` beforehand.
dataset = dataset.map(fn, lazy=True, batched=False)
# collate_fn
dataloader = Dataloader(dataset.with_format('torch'), collate_fn=collate_fn, num_workers=...) 
```

There are some minor decisions like whether a lazy map should be allowed before another map, but I think we can work it out later. The implementation can probably borrow from `IterableDataset`.
Issue: None batched `with_transform`, `set_transform`

Comment:
I like the idea of lazy map. On the other hand we should only have either lazy map or `with_transform` (not both). That's why I'd rather stick with `with_transform` for now (but maybe we can consider it for later major releases like `datasets` v2).

I understand the issue with `with_transform` and `with_format` being exclusive, maybe we can separate them: first transform, them format.

Finally I think what's also going to be important in the end will be the addition of multiprocessing to transforms
Issue: Unable to load audio_features from common_voice dataset

Comment:
Hi ! Feel free to access `batch["audio"]["array"]` and `batch["audio"]["sampling_rate"]` instead

`datasets` 1.16 introduced some changes in `common_voice` and now the `path` field is no longer a path to a local file (but rather the path to the file in the archive it's extracted from)
Issue: NonMatchingChecksumError for the CLUE:cluewsc2020, chid, c3 and tnews

Comment:
Seems like the issue still exists,:
`Downloading and preparing dataset clue/chid (download: 127.15 MiB, generated: 259.71 MiB, post-processed: Unknown size, total: 386.86 MiB) to /mnt/cache/tanhaochen/.cache/huggingface/datasets/clue/chid/1.0.0/e55b490cb7809dcd8db31b9a87119f2e2ec87cdc060da8a9ac070b070ca3e379...
Traceback (most recent call last):
  File "/mnt/cache/tanhaochen/PromptCLUE/test_datasets.py", line 3, in <module>
    cluewsc2020 = datasets.load_dataset("clue","chid")
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/load.py", line 1667, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/builder.py", line 593, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/builder.py", line 663, in _download_and_prepare
    verify_checksums(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://storage.googleapis.com/cluebenchmark/tasks/chid_public.zip']
`
Issue: [Audio] Allow resampling for audio datasets in streaming mode

Comment:
This requires implementing `cast_column` for iterable datasets, it could be a very nice addition !


It can also be useful to be able to disable the audio/image decoding for the dataset viewer (see PR https://github.com/huggingface/datasets/pull/3430) cc @severo 

EDIT: actually following https://github.com/huggingface/datasets/issues/3145 the dataset viewer might not need it anymore
Issue: [Audio] Allow resampling for audio datasets in streaming mode

Comment:
Just to clarify a bit. This feature is **always** needed when using the common voice dataset in streaming mode. So I think it's quite important
Issue: Add task tags for multimodal datasets

Comment:
The Hub pulls these tags from [here](https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts) (allows multimodal tasks) now, so I'm closing this issue.
Issue: add new field, and get errors

Comment:
Hi, 

could you please post this question on our [Forum](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests?
Issue: add new field, and get errors

Comment:
> Hi,
> 
> could you please post this question on our [Forum](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests?

ok.
Issue:  add one field  "example_id", but I can't see it in the "comput_loss" function

Comment:
Hi ! Your function looks fine, I used to map `squad` locally and it indeed added the `example_id` field correctly.

However I think that in the `compute_loss` method only a subset of the fields are available: the model inputs. Since `example_id` is not a model input (it's not passed as a parameter to the model), the data loader doesn't need to return it by default.

However you can disable this behavior by setting `remove_unused_columns` to `False` to your training arguments. In this case in `compute_loss` you will get the full item with all the fields.

Note that since the model doesn't take `example_id` as input, you will have to remove it from the inputs when `model(**inputs)` is called
Issue:  add one field  "example_id", but I can't see it in the "comput_loss" function

Comment:
Hmmm, it might be because the default data collator removes all the fields with `string` type:

https://github.com/huggingface/transformers/blob/4c0dd199c8305903564c2edeae23d294edd4b321/src/transformers/data/data_collator.py#L107-L112

I guess you also need a custom data collator that doesn't remove them.
Issue:  add one field  "example_id", but I can't see it in the "comput_loss" function

Comment:
I overwrite **get_train_dataloader**, and remove **_remove_unused_columns**, but it doesn't work.

```
    def get_train_dataloader(self) -> DataLoader:
        """
        Returns the training :class:`~torch.utils.data.DataLoader`.

        Will use no sampler if :obj:`self.train_dataset` does not implement :obj:`__len__`, a random sampler (adapted
        to distributed training if necessary) otherwise.

        Subclass and override this method if you want to inject some custom behavior.
        """
        if self.train_dataset is None:
            raise ValueError("Trainer: training requires a train_dataset.")

        train_dataset = self.train_dataset
        # if is_datasets_available() and isinstance(train_dataset, datasets.Dataset):
        #     train_dataset = self._remove_unused_columns(train_dataset, description="training")

        if isinstance(train_dataset, torch.utils.data.IterableDataset):
            if self.args.world_size > 1:
                train_dataset = IterableDatasetShard(
                    train_dataset,
                    batch_size=self.args.train_batch_size,
                    drop_last=self.args.dataloader_drop_last,
                    num_processes=self.args.world_size,
                    process_index=self.args.process_index,
                )

            return DataLoader(
                train_dataset,
                batch_size=self.args.train_batch_size,
                collate_fn=self.data_collator,
                num_workers=self.args.dataloader_num_workers,
                pin_memory=self.args.dataloader_pin_memory,
            )

        train_sampler = self._get_train_sampler()

        return DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
            pin_memory=self.args.dataloader_pin_memory,
        )
```
Issue:  add one field  "example_id", but I can't see it in the "comput_loss" function

Comment:
Hi, it works now, thank you.
1. **args.remove_unused_columns=False** and **training_args.remove_unused_columns=False**
2. overwrite **get_train_dataloader**, and remove **_remove_unused_columns**
3. add new fields, and can be got in **inputs**.
Issue: Failed to download species_800 from Google Drive zip file

Comment:
Hi,

the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?
Issue: Failed to download species_800 from Google Drive zip file

Comment:
> Hi,
> 
> the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

I have tried that many times with both load_dataset() and a browser almost simultaneously. The browser always works for me while load_dataset() fails.
Issue: Failed to download species_800 from Google Drive zip file

Comment:
@mariosasko 
> the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

I've tried yet again just a moment ago. This time I realize that, the step `(... post-processed: Unknown size, total: 20.89 MiB) to /Users/mike/.cache/huggingface/datasets/species800/species_800/1.0.0/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976...` and the one after seem unstable. If I want to retry, I will have to delete it (and probably other cache lock files). It **_sometimes_** works.

But I didn't try `download_mode="force_redownload"` yet.

Anyway, I suppose this isn't really a pressing issue for the time being, so I'm going to close this. Thank you.
Issue: Mirror the canonical datasets to the Hugging Face Hub

Comment:
I created a GitHub project to keep track of what needs to be done:
https://github.com/huggingface/datasets/projects/3

I also store my code in a (private for now) repository at https://github.com/huggingface/mirror_canonical_datasets_on_hub
Issue: Mirror the canonical datasets to the Hugging Face Hub

Comment:
I understand that the datasets are mirrored on the Hub now, right? Might I close @lhoestq @SBrandeis?
Issue: to_tf_dataset fails on TPU

Comment:
This might be related to https://github.com/tensorflow/tensorflow/issues/38762 , what do you think @Rocketknight1 ?
> Dataset.from_generator is expected to not work with TPUs as it uses py_function underneath which is incompatible with Cloud TPU 2VM setup. If you would like to read from large datasets, maybe try to materialize it on disk and use TFRecordDataest instead.
Issue: to_tf_dataset fails on TPU

Comment:
Hi @lhoestq @nbroad1881, I think it's very similar, yes. Unfortunately `to_tf_dataset` uses `tf.numpy_function` which can't be compiled - this is a necessary evil to load from the underlying Arrow dataset. We need to update the notebooks/examples to clarify that this won't work, or to identify a workaround. You may be able to get it to work on an actual cloud TPU VM, but those are quite new and we haven't tested it yet.
Issue: to_tf_dataset fails on TPU

Comment:
Thank you for the explanation. I didn't realize the nuances of `tf.numpy_function`. In this scenario, would it be better to use `export(format='tfrecord')` ?  It's not quite the same, but for very large datasets that don't fit in memory it looks like it is the only option. I haven't used `export` before, but I do recall reading that there are suggestions for how big and how many tfrecords there should be to not bottleneck the TPU. It might be nice if there were a way for the `export` method to split the files up into appropriate chunk sizes depending on the size of the dataset and the number of devices. And if that is too much, it would be nice to be able to specify the number of files that would be created when using `export`. Well... maybe the user should just do the chunking themselves and call `export` a bunch of times.  Whatever the case, you have been helpful. Thanks Tensorflow boy ;-)
Issue: to_tf_dataset fails on TPU

Comment:
Yeah, this is something we really should have a proper guide on. I'll make a note to test some things and make a 'TF TPU best practices' notebook at some point, but in the meantime I think your solution of exporting TFRecords will probably work.
Issue: Typing of Dataset.__getitem__ could be improved.

Comment:
Hi ! Thanks for the suggestion, I didn't know about this decorator.

If you are interesting in contributing, feel free to open a pull request to add the overload methods for each typing combination :) To assign you to this issue, you can comment `#self-assign` in this thread.

`Dataset.__getitem__` is defined right here: https://github.com/huggingface/datasets/blob/e6f1352fe19679de897f3d962e616936a17094f5/src/datasets/arrow_dataset.py#L1840
Issue: Integrate Polars library

Comment:
Polars has grown rapidly in popularity over the last year - could you consider integrating the Polars functionality again?

I don't think the "custom" implementation should be a barrier, it still conforms to the Arrow specification
Issue:  load JSON files, get the errors

Comment:
Hi ! The message you're getting is not an error. It simply says that your JSON dataset is being prepared to a location in `/root/.cache/huggingface/datasets`
Issue:  load JSON files, get the errors

Comment:
> 

but I want to load local JSON file by command
`python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/`

**squad-retrain-data/train-v2.0.json** is the local JSON file, how to load it and map it to a special structure?
Issue:  load JSON files, get the errors

Comment:
You can load it with `dataset = datasets.load_dataset('json', data_files=args.dataset)` as you said.
Then if you need to apply additional processing to map it to a special structure, you can use rename columns or use `dataset.map`. For more information, you can check the documentation here: https://huggingface.co/docs/datasets/process.html

Also feel free to share your `run.py` code so we can take a look
Issue:  load JSON files, get the errors

Comment:
```
# Dataset selection
    if args.dataset.endswith('.json') or args.dataset.endswith('.jsonl'):
        dataset_id = None
        # Load from local json/jsonl file
        dataset = datasets.load_dataset('json', data_files=args.dataset)
        # By default, the "json" dataset loader places all examples in the train split,
        # so if we want to use a jsonl file for evaluation we need to get the "train" split
        # from the loaded dataset
        eval_split = 'train'
    else:
        default_datasets = {'qa': ('squad',), 'nli': ('snli',)}
        dataset_id = tuple(args.dataset.split(':')) if args.dataset is not None else \
            default_datasets[args.task]
        # MNLI has two validation splits (one with matched domains and one with mismatched domains). Most datasets just have one "validation" split
        eval_split = 'validation_matched' if dataset_id == ('glue', 'mnli') else 'validation'
        # Load the raw data
        dataset = datasets.load_dataset(*dataset_id)
```

I want to load JSON squad dataset instead `dataset = datasets.load_dataset('squad')` to retrain the model.
Issue:  load JSON files, get the errors

Comment:
If your JSON has the same format as the SQuAD dataset, then you need to pass `field="data"` to `load_dataset`, since the SQuAD format is one big JSON object in which the "data" field contains the list of questions and answers.
```python
dataset = datasets.load_dataset('json', data_files=args.dataset, field="data")
```

Let me know if that helps :)
Issue:  load JSON files, get the errors

Comment:
Yes,  code works. but the format is not as expected.
```
dataset = datasets.load_dataset('json', data_files=args.dataset, field="data")
```
```
python3 run.py --do_train --task qa --dataset squad --output_dir ./re_trained_model/
```
************ train_dataset: Dataset({
    features: ['id', 'title', 'context', 'question', 'answers'],
    num_rows: 87599
})


```
python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/
```
************ train_dataset: Dataset({
    features: ['title', 'paragraphs'],
    num_rows: 442
})

I want the JSON to have the same format as before features. https://github.com/huggingface/datasets/blob/master/datasets/squad_v2/squad_v2.py is the script dealing with **squad** but how can I apply it by using JSON?
Issue:  load JSON files, get the errors

Comment:
Ok I see, you have the paragraphs so you just need to process them to extract the questions and answers. I think you can process the SQuAD-like data this way:
```python
def process_squad(articles):
    out = {
        "title": [],
        "context": [],
        "question": [],
        "id": [],
        "answers": [],
    }
    for title, paragraphs in zip(articles["title"], articles["paragraphs"]):
        for paragraph in paragraphs:
            for qa in paragraph["qas"]:
                out["title"].append(title)
                out["context"].append(paragraph["context"])
                out["question"].append(qa["question"])
                out["id"].append(qa["id"])
                out["answers"].append({
                    "answer_start": [answer["answer_start"] for answer in qa["answers"]],
                    "text": [answer["text"] for answer in qa["answers"]],
                })
    return out

dataset = dataset.map(process_squad, batched=True, remove_columns=["paragraphs"])
```

I adapted the code from [squad.py](https://github.com/huggingface/datasets/blob/master/datasets/squad/squad.py). The code takes as input a batch of articles (title + paragraphs) and gets all the questions and answers from the JSON structure.

The output is a dataset with `features: ['answers', 'context', 'id', 'question', 'title']`

Let me know if that helps !
Issue:  load JSON files, get the errors

Comment:
Yes, this works. But how to get the training output during training the squad by **Trainer** 
for example https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/trainer_qa.py 
I want the training inputs, labels, outputs for every epoch and step to produce the training dynamic graph
Issue:  load JSON files, get the errors

Comment:
I think you may need to implement your own Trainer, from the `QuestionAnsweringTrainer` for example.
This way you can have the flexibility of saving all the inputs/output used at each step
Issue:  load JSON files, get the errors

Comment:
> does there have any function to be overwritten to do this?

ok, I overwrote the compute_loss, thank you.
Issue: AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'

Comment:
Hi,

the fix was merged and will be available in the next release of `datasets`.
In the meantime, you can use it by installing `datasets` directly from master as follows:
```
pip install git+https://github.com/huggingface/datasets.git
```
Issue: Map function: Type error on iter #999

Comment:
Hi, thanks for reporting.

It would be really helpful if you could provide the actual code of the `text_numbers_to_int` function so we can reproduce the error.