SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Bug in wiki bio load

Comment:
+1, here's the error I got: 

```
>>> from datasets import load_dataset
>>>
>>> load_dataset("wiki_bio")
Downloading: 7.58kB [00:00, 4.42MB/s]
Downloading: 2.71kB [00:00, 1.30MB/s]
Using custom data configuration default
Downloading and preparing dataset wiki_bio/default (download: 318.53 MiB, generated: 736.94 MiB, post-processed: Unknown size, total: 1.03 GiB) to /home/jxm3/.cache/huggingface/datasets/wiki_bio/default/1.1.0/5293ce565954ba965dada626f1e79684e98172d950371d266bf3caaf87e911c9...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/load.py", line 1694, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/builder.py", line 595, in download_and_prepare
    self._download_and_prepare(
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/builder.py", line 662, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jxm3/.cache/huggingface/modules/datasets_modules/datasets/wiki_bio/5293ce565954ba965dada626f1e79684e98172d950371d266bf3caaf87e911c9/wiki_bio.py", line 125, in _split_generators
    data_dir = dl_manager.download_and_extract(my_urls)
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 308, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 251, in map_nested
    return function(data_struct)
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jxm3/.conda/envs/torch/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://drive.google.com/uc?export=download&id=1L7aoUXzHPzyzQ0ns4ApBbYepsjFOtXil
>>>
```
Issue: Bug in wiki bio load

Comment:
@alejandrocros and @lhoestq - you added the wiki_bio dataset in #1173. It doesn't work anymore. Can you take a look at this?
Issue: Bug in wiki bio load

Comment:
And if something is wrong with Google Drive, you could try to download (and collate and unzip) from here: https://github.com/DavidGrangier/wikipedia-biography-dataset
Issue: Bug in wiki bio load

Comment:
Hi ! Thanks for reporting. I've downloaded the data and concatenated them into a zip file available here: https://huggingface.co/datasets/wiki_bio/tree/main/data

I guess we can update the dataset script to use this zip file now :)
Issue: label information get lost after parquet serialization

Comment:
Hi ! We did a release of `datasets` today that may fix this issue. Can you try updating `datasets` and trying again ?

EDIT: the issue is still there actually

I think we can fix that by storing the Features in the parquet schema metadata, and then reload them when loading the parquet file
Issue: ConnectionError in IndicGLUE dataset

Comment:
@sahoodib, thanks for reporting.

Indeed, none of the data links appearing in the IndicGLUE website are working, e.g.: https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/soham-articles.tar.gz
```
<Error>
<Code>UserProjectAccountProblem</Code>
<Message>User project billing account not in good standing.</Message>
<Details>
The billing account for the owning project is disabled in state delinquent
</Details>
</Error>
```

We have contacted the data owners to inform them about their issue and ask them if they plan to fix it.
Issue: ConnectionError in IndicGLUE dataset

Comment:
Yesterday I resent a reminder email with more AI4Bharat-related people in the loop.

I also opened an issue in their repos:
- https://github.com/AI4Bharat/indicnlp_corpus/issues/14
- https://github.com/AI4Bharat/ai4bharat.org/issues/71
Issue: ConnectionError in IndicGLUE dataset

Comment:
We have received a reply from the authors reporting they have updated the URLs of their data files and opened a PR. See:
- #4978
Issue: Downloading Hugging Face Medical Dialog Dataset NonMatchingSplitsSizesError

Comment:
Hi @fabianslife, thanks for reporting.

I think you were using an old version of `datasets` because this bug was already fixed in version `1.13.0` (13 Oct 2021):
- Fix: 55fd140a63b8f03a0e72985647e498f1fc799d3f
- PR: #3046
- Issue: #2969 

Please, feel free to update the library: `pip install -U datasets`.
Issue: Dataset.from_pandas preserves useless index

Comment:
Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Issue: Cannot load ‘bookcorpusopen’

Comment:
The host of this copy of the dataset (https://the-eye.eu) is down and has been down for a good amount of time ([potentially months](https://www.reddit.com/r/Roms/comments/q82s15/theeye_downdied/))

Finding this dataset is a little esoteric, as the original authors took down the official BookCorpus dataset some time ago.

There are community-created versions of BookCorpus, such as the files hosted in the link below.
https://battle.shawwn.com/sdb/bookcorpus/

And more discussion here:
https://github.com/soskek/bookcorpus

Do we want to remove this dataset entirely?  There's a fair argument for this, given that the official BookCorpus dataset was taken down by the authors.   If not, perhaps can open a PR with the link to the community-created tar above and updated dataset description.
Issue: Cannot load ‘bookcorpusopen’

Comment:
Hi! The `bookcorpusopen` dataset is not working for the same reason as explained in this comment: https://github.com/huggingface/datasets/issues/3504#issuecomment-1004564980
Issue: Cannot load ‘bookcorpusopen’

Comment:
Hi @HUIYINXUE, it should work now that the data owners created a mirror server with all data, and we updated the URL in our library.
Issue: Integrate Milvus (pymilvus) library

Comment:
Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?
Issue: Integrate Milvus (pymilvus) library

Comment:
Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.
Issue: Integrate Milvus (pymilvus) library

Comment:
> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
Issue: DuplicatedKeysError when loading tweet_qa dataset

Comment:
Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
Issue: ImportError: cannot import name 'is_valid_waiter_error'

Comment:
Hi! I can't reproduce this error in Colab, but I'm assuming you are using Amazon SageMaker Studio Notebooks (you mention the `conda_pytorch_p36` kernel), so maybe @philschmid knows more about what might be causing this issue?
Issue: ImportError: cannot import name 'is_valid_waiter_error'

Comment:
Hey @mariosasko. Yes, I am using **Amazon SageMaker Studio Jupyter Labs**. However, I no longer need this notebook; but it would be nice to have this problem solved for others. So don't stress too much if you two can't reproduce error.
Issue: ImportError: cannot import name 'is_valid_waiter_error'

Comment:
Hey @danielbellhv, 

This issue might be related to Studio probably not having an up to date `botocore` and `boto3` version. I ran into this as well a while back. My workaround was 
```python
# using older dataset due to incompatibility of sagemaker notebook & aws-cli with > s3fs and fsspec to >= 2021.10
!pip install  "datasets==1.13" --upgrade
```

In `datasets` we use the latest `s3fs` and `fsspec` but aws-cli and notebook is not supporting this. You could also update the `aws-cli` and associated packages to get the latest `datasets` version
Issue: set_format("np") no longer works for Image data

Comment:
This error also propagates to jax and is even trickier to fix, since `.with_format(type='jax')` will use numpy conversion internally (and fail). For a three line failure:

```python
dataset = datasets.load_dataset("mnist")
dataset.set_format("jax")
X_train = dataset["train"]["image"]
```
Issue: set_format("np") no longer works for Image data

Comment:
Hi! We've recently introduced a new Image feature that yields PIL Images (and caches transforms on them) instead of arrays.

However, this feature requires a custom transform to yield np arrays directly:
```python
ddict = datasets.load_dataset("mnist")

def pil_image_to_array(batch):
     return {"image": [np.array(img) for img in batch["image"]]}  # or jnp.array(img) for Jax

ddict.set_transform(pil_image_to_array, columns="image", output_all_columns=True)
```

[Docs](https://huggingface.co/docs/datasets/master/process.html#format-transform) on `set_transform`.

Also, the approach proposed by @cgarciae is not the best because it loads the entire column in memory.

@albertvillanova @lhoestq WDYT? The Audio and the Image feature currently don't support the TF/Jax/PT Formatters, but for the Numpy Formatter maybe it makes more sense to return np arrays (and not a dict in the case of the Audio feature or a PIL Image object in the case of the Image feature).
Issue: set_format("np") no longer works for Image data

Comment:
Yes I agree it should return arrays and not a PIL image (and possible an array instead of a dict for audio data).
I'm currently finishing some code refactoring of the image and audio and opening a PR today. Maybe we can look into that after the refactoring
Issue: Specify the feature types of a dataset on the Hub without needing a dataset script

Comment:
After looking into this, discovered that this is already supported if the `dataset_infos.json` file is configured correctly! Here is a working example: https://huggingface.co/datasets/abidlabs/test-audio-13

This should be probably be documented, though.
Issue: Datasets created with `push_to_hub` can't be accessed in offline mode

Comment:
Thanks for reporting. I think this can be fixed by improving the `CachedDatasetModuleFactory` and making it look into the `parquet` cache directory (datasets from push_to_hub are loaded with the parquet dataset builder). I'll look into it
Issue: Datasets created with `push_to_hub` can't be accessed in offline mode

Comment:
We haven't had a chance to fix this yet. If someone would like to give it a try I'd be happy to give some guidance
Issue: Datasets created with `push_to_hub` can't be accessed in offline mode

Comment:
@lhoestq Do you have an idea of what changes need to be made to `CachedDatasetModuleFactory`? I would be willing to take a crack at it. Currently unable to train with datasets I have `push_to_hub` on a cluster whose compute nodes are not connected to the internet.

It looks like it might be this line:

https://github.com/huggingface/datasets/blob/0c1d099f87a883e52c42d3fd1f1052ad3967e647/src/datasets/load.py#L994

Which wouldn't pick up the stuff saved under `"datasets/allenai___parquet/*"`. Additionally, the datasets saved under `"datasets/allenai___parquet/*"` appear to have hashes in their name, e.g.  `"datasets/allenai___parquet/my_dataset-def9ee5552a1043e"`. This would not be detected by `CachedDatasetModuleFactory`, which currently looks for subdirectories

https://github.com/huggingface/datasets/blob/0c1d099f87a883e52c42d3fd1f1052ad3967e647/src/datasets/load.py#L995-L999
Issue: Datasets created with `push_to_hub` can't be accessed in offline mode

Comment:
`importable_directory_path` is used to find a **dataset script** that was previously downloaded and cached from the Hub

However in your case there's no dataset script on the Hub, only parquet files. So the logic must be extended for this case.

In particular I think you can add a new logic in the case where `hashes is None` (i.e. if there's no dataset script associated to the dataset in the cache).

In this case you can check directly in the in the datasets cache for a directory named `<namespace>__parquet` and a subdirectory named `<config_id>`. The config_id must match `{self.name.replace("/", "--")}-*`. 

In your case those two directories correspond to `allenai___parquet` and then `allenai--my_dataset-def9ee5552a1043e`

Then you can find the most recent version of the dataset in subdirectories (e.g. sorting using the last modified time of the `dataset_info.json` file).

Finally, we will need  return the module that is used to load the dataset from the cache. It is the same module than the one that would have been normally used if you had an internet connection.

At that point you can ping me, because we will need to pass all this:
- `module_path = _PACKAGED_DATASETS_MODULES["parquet"][0]`
- `hash` it corresponds the name of the directory that contains the .arrow file, inside `<namespace>__parquet/<config_id>`
- ` builder_kwargs = {"hash": hash, "repo_id": self.name, "config_id": config_id}`
and currently `config_id` is not a valid argument for a `DatasetBuilder`

I think in the future we want to change this caching logic completely, since I don't find it super easy to play with.
Issue: Datasets created with `push_to_hub` can't be accessed in offline mode

Comment:
Hi! Is there a workaround for the time being?
Like passing `data_dir` or something like that?

I would like to use [this diffuser example](https://github.com/huggingface/diffusers/tree/main/examples/unconditional_image_generation) on my cluster whose nodes are not connected to the internet. I have downloaded the dataset online form the login node.
Issue: Datasets created with `push_to_hub` can't be accessed in offline mode

Comment:
Hi ! Yes you can save your dataset locally with `my_dataset.save_to_disk("path/to/local")` and reload it later with `load_from_disk("path/to/local")`

(removing myself from assignees since I'm currently not working on this right now)
Issue: Allow loading community metrics from the hub, just like datasets

Comment:
Hi ! Thanks for your message :) This is a great idea indeed. We haven't started working on this yet though. For now I guess you can host your metric on the Hub (either with your model or your dataset) and use `hf_hub_download` to download it (docs [here](https://github.com/huggingface/huggingface_hub/blob/main/docs/hub/how-to-downstream.md#cached_download))
Issue: Allow loading community metrics from the hub, just like datasets

Comment:
Here's the code I used, in case it can be of help to someone else:
```python
import os, shutil
from huggingface_hub import hf_hub_download
def download_metric(repo_id, file_path):
    # repo_id: for models "username/model_name", for datasets "datasets/username/model_name"
    local_metric_path = hf_hub_download(repo_id=repo_id, filename=file_path)
    updated_local_metric_path = (os.path.dirname(local_metric_path) + os.path.basename(local_metric_path).replace(".", "_") + ".py")
    shutil.copy(local_metric_path, updated_local_metric_path)
    return updated_local_metric_path

metric = load_metric(download_metric(REPO_ID, FILE_PATH))
```
Issue: Task search function on hub not working correctly

Comment:
Because it has dots in some YAML keys, it can't be parsed and indexed by the back-end
Issue: Add PubMed Central Open Access dataset

Comment:
In the framework of BigScience:
- bigscience-workshop/data_tooling#121

I have created this dataset as a community dataset: https://huggingface.co/datasets/albertvillanova/pmc_open_access

However, I was wondering that it may be more appropriate to move it under an org namespace: `pubmed_central` or `pmc`
This way, we could add other datasets I'm also working on: Author Manuscript Dataset, Historical OCR Dataset, LitArch Open Access Subset.

What do you think? @lhoestq @mariosasko
Issue: Add PubMed Central Open Access dataset

Comment:
Why not ! Having them under such namespaces would also help people searching for this kind of datasets.
We can also invite people from pubmed at one point
Issue: `wiki_dpr` details for Open Domain Question Answering tasks

Comment:
Hi ! According to the DPR paper, the wikipedia dump is the one from Dec. 20, 2018.
Each instance contains a paragraph of at most 100 word, as well as the title of the wikipedia page it comes from and the DPR  embedding (a 768-d vector).
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
IMO, the data streaming test is good enough of a test that the dataset works correctly (assuming that we can more or less ensure that if streaming works then the non-streaming case will also work), so that for datasets that have a working dataset preview, we can remove the dummy data IMO. On the other hand, it seems like not all datasets have streaming enabled yet and for those datasets (if they are used a lot), I think it would be nice to continue testing some dummy data.

I don't really have an opinion regarding the JSON metadata as I don't know enough about it.
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
I don't know all the details, but generally I'd be in favor of unifying the metadata formats into YAML inside .md (and so deprecating the dataset_infos.json) 

(Ultimately the CI can run on "HuggingFace Actions" instead of on GitHub)
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
The dataset_infos.json file currently has these useful infos for each dataset configuration, that I think can be moved to the dataset tags:
- Size of the dataset in MB: download size, arrow file size, and total size (sum of download + arrow)
- Size of each split in MB and number of examples. Again this can be moved to the dataset tags
- Feature type of each column
- supported task templates (it defines what columns correspond to the features and labels for example)

But it also has this, which I'm not sure if it should be in the tags or not:
- Checksums of the downloaded files for integrity verifications

So ultimately this file could probably be deprecated in favor of having the infos in the tags.

> Also note that for generating both (dataset_infos.json file and dummy data), the entire dataset needs being downloaded. This can be an issue for huge datasets (like WIT, with 400 GB of data).

To get the exact number of examples and size in MB of the dataset, one needs to download and generate it completely. IMO these infos are very important when someone considers using a dataset. Though using streaming we could do some extrapolation to have approximate values instead.

For the integrity verifications we also need the number of examples and the checksums of the downloaded files, so it requires the dataset to be fully downloaded once. This can be optional though.

> IMO, the data streaming test is good enough of a test that the dataset works correctly (assuming that we can more or less ensure that if streaming works then the non-streaming case will also work)

I agree with this. Usually if a dataset works in streaming mode, then it works in non-streaming mode (the other way around is not true though).

> On the other hand, it seems like not all datasets have streaming enabled yet and for those datasets (if they are used a lot), I think it would be nice to continue testing some dummy data.

Yes indeed, or at least make sure that it was tested on the true data.
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
(note that if we wanted to display sizes, etc we could also pretty easily parse the `dataset_infos.json` on the hub side)
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
I agree that we can move the relevant parts of `dataset_infos.json` to the YAML tags.

> On the other hand, it seems like not all datasets have streaming enabled yet and for those datasets (if they are used a lot), I think it would be nice to continue testing some dummy data. <
> > Yes indeed, or at least make sure that it was tested on the true data.

I like the idea of testing streaming and falling back to the dummy data test if streaming does not work. Generating dummy data can be very tedious, so this would be a nice incentive for the contributors to make their datasets streamable.
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
About dummy data, please see e.g. this PR: https://github.com/huggingface/datasets/pull/3692/commits/62368daac0672041524a471386d5e78005cf357a
- I updated the previous dummy data: I just had to rename the file and its directory
  - the dummy data zip contains only a single file: `pubmed22n0001.xml.gz`

Then I discover it fails: https://app.circleci.com/pipelines/github/huggingface/datasets/9800/workflows/173a4433-8feb-4fc6-ab9e-59762084e3e1/jobs/60437
```
No such file or directory: '.../dummy_data/pubmed22n0002.xml.gz'
```
- it needs dummy data for all the 1114 files: 
  `_URLs = [f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n{i:04d}.xml.gz" for i in range(1, 1115)]`
- this confirms me that it never passed the test: these dummy data files were not present before my PR
- therefore, is it really useful the data test if we just ignore it when it does not pass?

In relation with JSON metadata, I'm generating the file for `pubmed` (see above) in a GCP instance: it's running for more than 3 hours and only 9 million examples generated so far (before my PR, it had 32 million, now it has more).
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
I mention in https://github.com/huggingface/datasets-server/wiki/Preliminary-design that the future "datasets server" could be in charge of generating both the dummy data and the dataset-info.json file if required (or their equivalent).
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
Hi ! I think dummy data generation is out of scope for the datasets server, since it's about generating the original data files.

That would be amazing to have it generate the dataset_infos.json though !
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
From some offline discussion with @mariosasko and especially for vision datasets, we'll probably not require dummy data anymore and use streaming instead :) This will make adding a new dataset much easier.
This should also make sure that streaming works as expected directly in the CI, without having to check the dataset viewer once the PR is merged