SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Allow downloading just some columns of a dataset

Comment:
Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.
Issue: Multiprocessing with FileLock fails in python 3.9

Comment:
Closing this one because it must be used this way actually:
```python
def main():
    with FileLock("tmp.lock"):
        with Pool(2) as pool:
            pool.map(run, range(2))

if __name__ == "__main__":
    main()
```
Issue: ImageFolder with Grayscale images dataset

Comment:
Hi! Replacing:
```python
transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")
```

with:
```python
def transform_func(examples):
    examples["image"] = [transforms(img).to("cuda") for img in examples["image"]]
    return examples

transformed_dataset = dataset.with_transform(transform_func)
```
should fix the issue. `datasets` doesn't support chaining of transforms (you can think of `set_format`/`with_format` as a predefined transform func for `set_transform`/`with_transforms`), so the last transform (in your case, `set_format`) takes precedence over the previous ones (in your case `with_format`). And the PyTorch formatter is not supported by the Image feature, hence the error (adding support for that is on our short-term roadmap).
Issue: ImageFolder with Grayscale images dataset

Comment:
Ok thanks a lot for the code snippet!

I love the way `datasets` is easy to use but it made it really long to pre-process all the images (400.000 in my case) before training anything. `ImageFolder` from pytorch is faster in my case but force me to have the images on my local machine.

I don't know how to speed up the process without switching to `ImageFolder` :smile:
Issue: ImageFolder with Grayscale images dataset

Comment:
You can pass `ignore_verifications=True` in `load_dataset` to skip checksum verification, which takes a lot of time if the number of files is large. We will consider making this the default behavior.
Issue: Unable to view the dataset and loading the same dataset throws the error - ArrowInvalid: Exceeded maximum rows

Comment:
It seems that train.json is not a valid JSON Lines file: it has several JSON objects in the first line (the 915th character in the first line starts a new object, and there's no "\n")

You need to have one JSON object per line
Issue: Unable to view the dataset and loading the same dataset throws the error - ArrowInvalid: Exceeded maximum rows

Comment:
I'm closing this issue.

@Pavithree, please, feel free to re-open it if fixing the JSON file does not solve it.
Issue: push to hub fails with huggingface-hub 0.5.0

Comment:
Hi ! Indeed there was a breaking change in `huggingface_hub` 0.5.0 in `HfApi.create_repo`, which is called here in `datasets` by passing the org name in both the `repo_id` and the `organization` arguments:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

I think we should fix that in `huggingface_hub`, will keep you posted. In the meantime please use `huggingface_hub` 0.4.0
Issue: push to hub fails with huggingface-hub 0.5.0

Comment:
I'll be sending a fix for this later today on the `huggingface_hub` side.

The error would be converted to a `FutureWarning` if `datasets` uses kwargs instead of positional, for example here: 

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

to be:

``` python
            api.create_repo(
                name=dataset_name,
                token=token,
                repo_type="dataset",
                organization=organization,
                private=private,
            )
```

But `name` and `organization` are deprecated in `huggingface_hub=0.5`, and people should pass `repo_id='org/name` instead. Note that `repo_id` was introduced in 0.5 and if `datasets` wants to support older `huggingface_hub` versions (which I encourage it to do), there needs to be a helper function to do that. It can be something like:


```python
def create_repo(
    client,
    name: str,
    token: Optional[str] = None,
    organization: Optional[str] = None,
    private: Optional[bool] = None,
    repo_type: Optional[str] = None,
    exist_ok: Optional[bool] = False,
    space_sdk: Optional[str] = None,
) -> str:
    try:
        return client.create_repo(
            repo_id=f"{organization}/{name}",
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
    except TypeError:
        return client.create_repo(
            name=name,
            organization=organization,
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
```

in a `utils/_fixes.py` kinda file and and be used internally.

I'll be sending a patch to `huggingface_hub` to convert the error reported in this issue to a `FutureWarning`.
Issue: push to hub fails with huggingface-hub 0.5.0

Comment:
We can definitely change `push_to_hub` to use `repo_id` in `datasets` and require `huggingface_hub>=0.5.0`.

Let me open a PR :)
Issue: Add time series data - stock market

Comment:
@INF800 happy to add this dataset! I will try to set a PR by the end of the day... if you can kindly point me to the dataset? Also, note we have a bunch of time series datasets checked in e.g. `electricity_load_diagrams` or `monash_tsf`, and ideally this dataset could also be in a similar format.
Issue: Add time series data - stock market

Comment:
Thankyou. This is how raw data looks like before cleaning for an individual stocks:

1. https://github.com/INF800/marktech/tree/raw-data/f/data/raw
2. https://github.com/INF800/marktech/tree/raw-data/t/data/raw
3. https://github.com/INF800/marktech/tree/raw-data/rdfn/data/raw
4. https://github.com/INF800/marktech/tree/raw-data/irbt/data/raw
5. https://github.com/INF800/marktech/tree/raw-data/hll/data/raw
6. https://github.com/INF800/marktech/tree/raw-data/infy/data/raw
7. https://github.com/INF800/marktech/tree/raw-data/reli/data/raw
8. https://github.com/INF800/marktech/tree/raw-data/hdbk/data/raw

> Scraping is automated using GitHub Actions. So, everyday we will see a new file added in the above links.

I can rewrite the cleaning scripts to make sure it fits HF dataset standards. (P.S I am very much new to HF dataset)

The data set above can be converted into univariate regression / multivariate regression / sequence to sequence generation dataset etc. So, do we have some kind of transformation modules that will read the dataset as some type of dataset (`GenericTimeData`) and convert it to other possible dataset relating to a specific ML task. **By having this kind of transformation module, I only have to add data once** and use transformation module whenever necessary

Additionally, having some kind of versioning for the dataset will be really helpful because it will keep on updating - especially time series datasets
Issue: Add time series data - stock market

Comment:
thanks @INF800 I'll have a look. I believe it should be possible to incorporate this into the time-series format.
Issue: Add time series data - stock market

Comment:
@INF800 yes I am aware of the review repository and paper which is more or less a collection of abstracts etc. I am working on a unified library of implementations of these papers together with datasets to be then able to compare/contrast and build upon the research etc. but I am not ready to share them publicly just yet.

In any case regarding your dataset at the moment its seems from looking at the csv files, its mixture of textual and numerical data, sometimes in the same column etc. As you know, for time series models we would need just numeric data so I would need your help in disambiguating the dataset you have collected and also perhaps starting with just numerical data to start with... 

Do you think you can make a version with just numerical data?
Issue: Add time series data - stock market

Comment:
> @INF800 yes I am aware of the review repository and paper which is more or less a collection of abstracts etc. I am working on a unified library of implementations of these papers together with datasets to be then able to compare/contrast and build upon the research etc. but I am not ready to share them publicly just yet.
> 
> In any case regarding your dataset at the moment its seems from looking at the csv files, its mixture of textual and numerical data, sometimes in the same column etc. As you know, for time series models we would need just numeric data so I would need your help in disambiguating the dataset you have collected and also perhaps starting with just numerical data to start with...
> 
> Do you think you can make a version with just numerical data?

Will share the numeric data and conversion script within end of this week. 

I am on a business trip currently - it is in my desktop.
Issue: How can I download only the train and test split for full numbers using load_dataset()? 

Comment:
Hi! Can you please specify the full name of the dataset? IIRC `full_numbers` is one of the configs of the `svhn` dataset, and its generation is slow due to data being stored in binary Matlab files. Even if you specify a specific split, `datasets` downloads all of them, but we plan to fix that soon and only download the requested split.

If you are in a hurry, download the `svhn` script [here](`https://huggingface.co/datasets/svhn/blob/main/svhn.py`), remove [this code](https://huggingface.co/datasets/svhn/blob/main/svhn.py#L155-L162), and run:
```python
from datasets import load_dataset
dset = load_dataset("path/to/your/local/script.py", "full_numbers")
```

And to make loading easier in Colab, you can create a dataset repo on the Hub and upload the script there. Or push the script to Google Drive and mount the drive in Colab.
Issue: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)

Comment:
Hi @andreybond, thanks for reporting.

Unfortunately, I'm not able to able to reproduce your issue:
```python
In [4]: from datasets import load_dataset
   ...: datasets = load_dataset("nielsr/XFUN", "xfun.ja")

In [5]: datasets
Out[5]: 
DatasetDict({
    train: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 194
    })
    validation: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 71
    })
})
```

The only reason I can imagine this issue may arise is if your default encoding is not "UTF-8" (and it is ASCII instead). This is usually the case on Windows machines; but you say your environment is a Linux machine. Maybe you change your machine default encoding?

Could you please check this?
```python
In [6]: import sys

In [7]: sys.getdefaultencoding()
Out[7]: 'utf-8'
```
Issue: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)

Comment:
I opened a PR in the original dataset loading script:
-  microsoft/unilm#677

and fixed the corresponding dataset script on the Hub:
- https://huggingface.co/datasets/nielsr/XFUN/commit/73ba5e026621e05fb756ae0f267eb49971f70ebd
Issue: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)

Comment:
import sys
sys.getdefaultencoding()

returned: 'utf-8'

---------------------

I've just cloned master branch - your fix works! Thank you!
Issue: Add support for streaming Zarr stores for hosted datasets

Comment:
Hi @jacobbieker, thanks for your request and study of possible alternatives.

We are very interested in finding a way to make `datasets` useful to you.

Looking at the Zarr docs, I saw that among its storage alternatives, there is the ZIP file format: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore

This might be convenient for many reasons:
- On the one hand, we avoid the Git issue with huge number of small files: chunks files are compressed into a single ZIP file
- On the other hand, the ZIP file format is specially suited for streaming data because it allows random access to its component files (i.e. it supports random access to its chunks)

Anyway, I think that a Python loading script will be necessary: you need to implement additional logic to select certain chunks (based on date or other criteria).

Please, let me know if this makes sense to you.
Issue: Add support for streaming Zarr stores for hosted datasets

Comment:
Ah okay, I missed the option of zip files for zarr, I'll try that with our repos and see if it works! Thanks a lot!
Issue: Add support for streaming Zarr stores for hosted datasets

Comment:
On behalf of the Zarr developers, let me say THANK YOU for working to support Zarr on HF! 🙏  Zarr is a 100% open-source and community driven project (fiscally sponsored by NumFocus). We see it as an ideal format for ML training datasets, particularly in scientific domains.

I think the solution of zipping the Zarr store is a reasonable way to balance the constraints of Git LFS with the structure of Zarr.

It would be amazing to get something on the [Hugging Face Datasets Docs](https://huggingface.co/docs/datasets/index) about how to best work with Zarr. Let me know if there's a way I could help with that effort.
Issue: Add support for streaming Zarr stores for hosted datasets

Comment:
Also just noting here that I was able to lazily open @jacobbieker's dataset over the internet from HF hub 🚀 !

```python
import xarray as xr
url = "https://huggingface.co/datasets/openclimatefix/mrms/resolve/main/data/2016_001.zarr.zip"
zip_url = 'zip:///::' + url
ds = xr.open_dataset(zip_url, engine='zarr', chunks={})
```

<img width="740" alt="image" src="https://user-images.githubusercontent.com/1197350/164508663-bc75cdc0-734d-44f4-9562-2877ecfdf433.png">
Issue: Add support for streaming Zarr stores for hosted datasets

Comment:
I'm still messing around with that dataset, so the data might have moved. I currently have each year of MRMS precipitation rate data as it's own zarr, but as they are quite large (on order of 100GB each) I'm working to split them into single days, and as such they are still being moved around, I was just trying to get a proof of concept working originally.
Issue: Add support for streaming Zarr stores for hosted datasets

Comment:
I've mostly finished rearranging the data now and uploading some more, so this works now:
```python
import datasets
ds = datasets.load_dataset("openclimatefix/mrms", streaming=True, split="train")
item = next(iter(ds))
print(item.keys())
print(item["timestamp"])
```

The MRMS data now goes most of 2016-2022, with quite a few gaps I'm working on filling in
Issue: elena-soare/crawled-ecommerce: missing dataset

Comment:
By the way, the error on our part is due to the huge size of every row (~90MB). The dataset viewer does not support such big dataset rows for the moment.
Anyway, we're working to give a hint about this in the dataset viewer.
Issue: elena-soare/crawled-ecommerce: missing dataset

Comment:
Fixed. See https://huggingface.co/datasets/elena-soare/crawled-ecommerce/viewer/elena-soare--crawled-ecommerce/train.

<img width="1552" alt="Capture d’écran 2022-04-12 à 11 23 51" src="https://user-images.githubusercontent.com/1676121/162929722-2e2b80e2-154a-4b61-87bd-e341bd6c46e6.png">

Thanks for reporting!
Issue: Build a Dataset One Example at a Time Without Loading All Data Into Memory

Comment:
Hi! Yes, the problem with `add_item` is that it keeps examples in memory, so you are left with these options:
* writing a dataset loading script in which you iterate over `custom_example_dict_streamer` and yield the examples (in `_generate examples`)
* storing the data in a JSON/CSV/Parquet/TXT file and using `Dataset.from_{format}`
* using `add_item` + `save_to_disk` on smaller chunks: 
    ```python
    from datasets import Dataset, concatenate_datasets
    MAX_SAMPLES_IN_MEMORY = 1000
    samples_in_dset = 0
    dset = Dataset.from_dict({"col1": [], "col2": []})  # empty dataset
    path_to_save_dir = "path/to/save/dir"
    num_chunks = 0
    for example_dict in custom_example_dict_streamer("/path/to/raw/data"):
        dset = dset.add_item(example_dict)
        samples_in_dset += 1
        if samples_in_dset == MAX_SAMPLES_IN_MEMORY:
            samples_in_dset = 0
            dset.save_to_disk(f"{path_to_save_dir}{num_chunks}")
            num_chunks =+ 1
            dset = Dataset.from_dict({"col1": [], "col2": []})  # empty dataset
    if samples_in_dset > 0:
        dset.save_to_disk(f"{path_to_save_dir}{num_chunks}")
        num_chunks =+ 1
    loaded_dsets = []  # memory-mapped
    for chunk_num in range(num_chunks):
        dset = Dataset.load_from_disk(f"{path_to_save_dir}{chunk_num}") 
        loaded_dsets.append(dset)
    final_dset = concatenate_datasets(dset)
    ```
    If you still have issues with this approach, you can try to delete unused datasets with `gc.collect()` to free some memory.
Issue: Dataset viewer issue for McGill-NLP/feedbackQA

Comment:
Hi @cslizc, thanks for reporting.

I have just forced the refresh of the corresponding cache and the preview is working now.
Issue: datasets.set_progress_bar_enabled(False) not working in datasets v2

Comment:
Now, I can't find any reference to set_progress_bar_enabled in the code.

I think it have been deleted
Issue: datasets.set_progress_bar_enabled(False) not working in datasets v2

Comment:
Hi @virilo,

Please note that since `datasets` version 2.0.0, we have aligned with `transformers` the management of the progress bar (among other things):
- #3897

Now, you should update your code to use `datasets.logging.disable_progress_bar`.

You have more info in our docs: [Logging methods](https://huggingface.co/docs/datasets/package_reference/logging_methods)
Issue: datasets.set_progress_bar_enabled(False) not working in datasets v2

Comment:
One important thing for beginner like me is: from datasets.utils.logging import disable_progress_bar
Do not forget the 'utils' or you will waste a long time like me....
Issue: Errors in `Train with Datasets` Tensorflow code section on Huggingface.co

Comment:
Hi @blackhat-coder, thanks for reporting.

Please note that the `transformers` library updated their data collators API last year (version 4.10.0):
- huggingface/transformers#13105

now requiring to pass `return_tensors` argument at Data Collator instantiation.

And therefore, we also updated in the `datasets` library documentation all the examples using `transformers` data collators.

If you would like to follow our examples, please update your installed `transformers` version:
```
pip install -U transformers
```
Issue: NonMatchingChecksumError for downloading conll2012_ontonotesv5 dataset

Comment:
Hi @richarddwang,


Indeed, we have recently updated the loading script of that dataset (and fixed that bug as well):
- #4002

That fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by:
- installing `datasets` from our GitHub repo:
```bash
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
- forcing the data files to be redownloaded
```python
ds = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test", download_mode="force_redownload")
```

Feel free to re-open this issue if the problem persists. 

Duplicate of:
- #4031
Issue: Add CCAgT dataset

Comment:
Awesome ! Let us know if you have questions or if we can help ;) I'm assigning you

PS: if possible, please try to not use Google Drive links in your dataset script, since Google Drive has download quotas and is not always reliable.
Issue: Add CCAgT dataset

Comment:
HI, I was waiting to come out in the second version to do the implementation.

- Paper: https://dx.doi.org/10.2139/ssrn.4126881
- Data: [Data mendelay](http://doi.org/10.17632/wg4bpm33hj.2)
Issue: Error in google/xtreme_s dataset card

Comment:
Hi @wranai, thanks for reporting.

Please note that the information about language families and groups is taken form the original paper: [XTREME-S: Evaluating Cross-lingual Speech Representations](https://arxiv.org/abs/2203.10752).

If that information is wrong, feel free to contact the paper's authors to suggest that correction.

Just note that Hungarian language (contrary to their geographically surrounding neighbor languages) belongs to the Uralic (languages) family, together with (among others) Finnish, Estonian, some other languages in northern regions of Scandinavia...
Issue: Loading issue for xuyeliu/notebookCDG dataset

Comment:
Hi @Jun-jie-Huang,

As the error message says, ".pkl" data files are not supported.

If you would like to share your dataset on the Hub, you would need:
- either to create a Python loading script, that loads the data in any format
- or to transform your data files to one of the supported formats (listed in the error message above: CSV, JSON, Parquet, TXT,...)

You can find the details in our docs: 
- How to share a dataset: https://huggingface.co/docs/datasets/share
- How to create a dataset loading script: https://huggingface.co/docs/datasets/dataset_script

Feel free to re-open this issue and ping us if you need further assistance.
Issue: Loading mozilla-foundation/common_voice_7_0 dataset failed

Comment:
Hi @aapot, thanks for reporting.

We are investigating the cause of this issue. We will keep you informed.
Issue: Loading mozilla-foundation/common_voice_7_0 dataset failed

Comment:
When making HTTP request from code line:
```
response = requests.get(f"{_API_URL}/bucket/dataset/{path}/{use_cdn}", timeout=10.0).json()
```
it cannot be decoded to JSON because it raises a 404 Not Found error.

The request is fixed if removing the `/{use_cdn}` from the URL.

Maybe there was a change in the Common Voice API?

CC: @anton-l @patrickvonplaten @polinaeterna
Issue: Loading mozilla-foundation/common_voice_7_0 dataset failed

Comment:
I have also made the hotfix for all the rest of Common Voice script versions: 8.0, 6.1, 6.0,..., 1.0
Issue: Loading mozilla-foundation/common_voice_7_0 dataset failed

Comment:
Hi @ngoquanghuy99! The dataset should load fine if you go through the following steps:

1. Go to https://huggingface.co/datasets/mozilla-foundation/common_voice_9_0 and click "Access repository" if you see a message about sharing your contact information with Mozilla Foundation at the top of the page. If you've already done that then skip to step 2.
2. Run the command `huggingface-cli login` in your terminal or notebook to authenticate your machine.
3. Load the dataset with `use_auth_token=True`:
```python
from datasets import load_dataset

dataset = load_dataset("mozilla-foundation/common_voice_9_0", "ab", use_auth_token=True)
```
Issue: Loading cnn_dailymail dataset failed

Comment:
Hi @Arij-Aladel, thanks for reporting.

This issue was already reported 
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it in our 2.0.0 release. See:
- #3787 

Please, update your `datasets` version:
```
pip install -U datasets
```
and retry loading the dataset by forcing its redownload:
```python
dataset = load_dataset("cnn_dailymail", "3.0.0", download_mode="force_redownload")
```