SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

Comment:
Hi! This issue stems from the fact that `xwalk`, which is a streamable version of `os.walk`, doesn't support the `topdown` param due to `fsspec`'s `walk` also not supporting it, so fixing this issue could be tricky.  

@MalakhovIlyaPavlovich You can avoid the error by tweaking your data processing and not using this param. (and `Path.rename`, which also cannot be streamed)
Issue: Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

Comment:
@mariosasko thank you for your reply. I couldn't reproduce error showed by @severo either on Ubuntu 20.04.3 LTS, Windows 10 and Google Colab environments. But trying to avoid using os.walk(topdown=False) and Path.rename(), In _split_generators I replaced
```
def decode_file_names(folder):
    for root, dirs, files in os.walk(folder, topdown=False):
        root = Path(root)
        for file in files:
            old_name = root / Path(file)
            new_name = root / Path(
                file.encode('cp437').decode('cp866'))
            old_name.rename(new_name)
        for dir in dirs:
            old_name = root / Path(dir)
            new_name = root / Path(dir.encode('cp437').decode('cp866'))
            old_name.rename(new_name)

folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
decode_file_names(folder)
```
by
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
if not is_url(zip_file):
    folder = extract(zip_file)
else:
    folder = None
```
and now everything works well except data viewer for "raw_txt" subset: dataset preview on hub shows "No data.". As far as I understand dl_manager.download returns original URL when we are calling datasets.get_dataset_split_names and my suspicions are that dataset viewer can do smth similar. I couldn't find information about how it works. I would be very grateful, if you could tell me how to fix this)
Issue: Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

Comment:
This is what I get when I try to stream the `raw_txt` subset:
```python
>>> dset = load_dataset("MalakhovIlya/RuREBus", "raw_txt", split="raw_txt", streaming=True)
>>> next(iter(dset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
```
So there is a bug in your script.
Issue: Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

Comment:
streaming=True helped me to find solution. I fixed
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
folder = extract(zip_file)
```
by 
```
folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
path = os.path.join(folder, 'MED_txt/unparsed_txt')
for root, dirs, files in os.walk(path):
    decoded_root_name = Path(root).name.encode('cp437').decode('cp866')
```
@mariosasko thank you for your help :)
Issue: ELI5 supporting documents

Comment:
Hi ! Please post your question on the [forum](https://discuss.huggingface.co/), more people will be able to help you there ;)
Issue: HANS dataset preview broken

Comment:
Hi! I've opened a PR that should make this dataset stremable. You can test it as follows:
```python
from datasets import load_dataset
dset = load_dataset("hans", split="train", streaming=True, revision="49decd29839c792ecc24ac88f861cbdec30c1c40")
```

@severo The current script doesn't throw an error in normal mode (only in streaming mode) on my local machine or in Colab. Can you update your installation of `datasets` and see if that fixes the issue?
Issue: HANS dataset preview broken

Comment:
Thanks for this. It works well, thanks! The dataset viewer is using https://github.com/huggingface/datasets/releases/tag/2.0.0, I'm eager to upgrade to 2.0.1 😉
Issue: Image decoding often fails when  transforming Image datasets

Comment:
A quick hack I have found is that we can call the image first before running the transforms and it makes sure the image is decoded before being passed on.

For this I just needed to add `example['img'] = example['img']` to the top of my `generate_flipped_data` function, defined above, so that image decode in invoked.

After this minor change this function works:
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    example['img'] = example['img']  # <<< This is the only change
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example
```
Issue: Image decoding often fails when  transforming Image datasets

Comment:
Hi @RafayAK, thanks for reporting.

Current implementation of the Image feature performs the decoding only if the "img" field is accessed by the mapped function.

In your original `generate_flipped_data` function:
- it only accesses the "img" field (and thus performs decoding) if `rng.random() > p`;
- on the other hand, for the cases where `rng.random() <= p`, the "img" field is not accessed and thus no decoding is performed for those examples

By adding the code line `example['img'] = example['img']`, you make sure the "img" field is accessed in all cases, and the decoding is done for all examples.

Also note that there is a little bug in your implementation: `p` is not the probability of flipping, but the probability of not-flipping; the larger is `p`, the smaller is the probability of flipping.

Some refactoring (fixing also `p`):
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down.

    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    do_flip = rng.random() < p  # Note the "<" sign here instead of ">"
    example['img'] = example['img'].transpose(1) if do_flip else example['img']  # Note "img" is always accessed
    example['is_flipped'] = 1 if do_flip else 0
    return example
Issue: Image decoding often fails when  transforming Image datasets

Comment:
@albertvillanova Thanks for letting me know this is intended behavior. The docs are severely lacking on this, if I hadn't posted this here I would have never found out how I'm actually supposed to modify images in a Dataset object.
Issue: Image decoding often fails when  transforming Image datasets

Comment:
@albertvillanova Secondly if you check the error message it shows that around 1999 images were successfully created, I'm pretty sure some of them were also flipped during the process. Back to my main contention, sometimes the decoding takes place other times it fails. 

I suppose to run `map` on any dataset all the examples should be invoked even if on some of them we end up doing nothing, is that right?
Issue: Image decoding often fails when  transforming Image datasets

Comment:
Hi @RafayAK! I've opened a PR with the fix, which adds a fallback to reattempt casting to PyArrow format with a more robust (but more expensive) procedure if the first attempt fails. Feel free to test it by installing `datasets` from the PR branch with the following command:
```
pip install git+https://github.com/huggingface/datasets.git@fix-4124
```
Issue: Image decoding often fails when  transforming Image datasets

Comment:
@mariosasko Thanks a lot for looking into this, now the `map` function at least behaves as one would expect a function to behave. 

Looking forward to exploring Hugging Face more and even contributing 😃.

```bash
 $ conda list | grep datasets
datasets                  2.0.1.dev0               pypi_0    pypi

```

```python
def preprocess_data(dataset):
    """
    Helper funtion to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset


def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)
```

The output now show the function was applied successfully:
``` bash
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Downloading builder script: 5.61kB [00:00, 3.16MB/s]                   
Downloading metadata: 4.21kB [00:00, 2.56MB/s]                   
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
100%|██████████| 10000/10000 [00:01<00:00, 5149.15ex/s]
```
Issue: Building C4 takes forever

Comment:
Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...: 
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.
Issue: medical_dialog zh has very slow _generate_examples

Comment:
Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance.
Issue: medical_dialog zh has very slow _generate_examples

Comment:
Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this. 
@albertvillanova please let me know if I am doing something unnecessary or time consuming.
Issue: medical_dialog zh has very slow _generate_examples

Comment:
Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```
Issue: datasets.load_metric can not load a local metirc

Comment:
Hello, could you tell me how this issue can be fixed? I'm coming across the same issue.
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
Hi @arymbe, thanks for reporting.

Unfortunately, I'm not able to reproduce your problem.

Could you please write the complete stack trace? That way we will be able to see which package originates the exception.
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
Hello, thank you for your fast replied. this is the complete error that I got

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

Input In [27], in <module>
----> 1 from datasets import load_dataset

venv/lib/python3.8/site-packages/datasets/__init__.py:39, in <module>
     37 from .arrow_dataset import Dataset, concatenate_datasets
     38 from .arrow_reader import ReadInstruction
---> 39 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     40 from .combine import interleave_datasets
     41 from .dataset_dict import DatasetDict, IterableDatasetDict

venv/lib/python3.8/site-packages/datasets/builder.py:40, in <module>
     32 from .arrow_reader import (
     33     HF_GCP_BASE_URL,
     34     ArrowReader,
   (...)
     37     ReadInstruction,
     38 )
     39 from .arrow_writer import ArrowWriter, BeamWriter
---> 40 from .data_files import DataFilesDict, sanitize_patterns
     41 from .dataset_dict import DatasetDict, IterableDatasetDict
     42 from .features import Features

venv/lib/python3.8/site-packages/datasets/data_files.py:297, in <module>
    292     except FileNotFoundError:
    293         raise FileNotFoundError(f"The directory at {base_path} doesn't contain any data file") from None
    296 def _resolve_single_pattern_in_dataset_repository(
--> 297     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    298     pattern: str,
    299     allowed_extensions: Optional[list] = None,
    300 ) -> List[PurePath]:
    301     data_files_ignore = FILES_TO_IGNORE
    302     fs = HfFileSystem(repo_info=dataset_info)

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
This is weird... It is long ago that the package `huggingface_hub` has a submodule called `hf_api`.

Maybe you have a problem with your installed `huggingface_hub`...

Could you please try to update it?
```shell
pip install -U huggingface_hub
```
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
Yap, I've updated several times. Then, I've tried numeral combination of datasets and huggingface_hub versions. However, I think your point is right that there is a problem with my huggingface_hub installation. I'll try another way to find the solution. I'll update it later when I get the solution. Thank you :)
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
I'm sorry I can't reproduce your problem.

Maybe you could try to create a new Python virtual environment and install all dependencies there from scratch. You can use either:
- Python venv: https://docs.python.org/3/library/venv.html
- or conda venv (if you are using conda): https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
Facing the same issue.

Response from `pip show datasets`
```
Name: datasets
Version: 1.15.1
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: aiohttp, dill, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, requests, tqdm, xxhash
Required-by: lm-eval
```

Response from `pip show huggingface_hub`

```
Name: huggingface-hub
Version: 0.8.1
Summary: Client library to download and publish models, datasets and other repos on the huggingface.co hub
Home-page: https://github.com/huggingface/huggingface_hub
Author: Hugging Face, Inc.
Author-email: julien@huggingface.co
License: Apache
Location: /usr/local/lib/python3.8/dist-packages
Requires: filelock, packaging, pyyaml, requests, tqdm, typing-extensions
Required-by: datasets
```

response from `datasets-cli env`

```
Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 5, in <module>
    from datasets.commands.datasets_cli import main
  File "/usr/local/lib/python3.8/dist-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/usr/local/lib/python3.8/dist-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/__init__.py", line 105, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api
```
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
A workaround: 
I changed lines around Line 125 in `__init__.py` of `huggingface_hub` to something like
```
__getattr__, __dir__, __all__ = _attach(
    __name__,
    submodules=['hf_api'],
```
and it works ( which gives `datasets` direct access to `huggingface_hub.hf_api` ).
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
I was getting the same issue. After trying a few versions, following combination worked for me.
dataset==2.3.2
huggingface_hub==0.7.0

In another environment, I just installed latest repos from pip through `pip install -U transformers datasets tokenizers evaluate`, resulting in following versions. This also worked. Hope it helps someone. 

datasets-2.3.2 evaluate-0.1.2 huggingface-hub-0.8.1 responses-0.18.0 tokenizers-0.12.1 transformers-4.20.1
Issue: AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

Comment:
(For layoutlmv3 fine-tuning) In my case, modifying `requirements.txt` as below worked.

- python = 3.7

```
datasets==2.3.2
evaluate==0.1.2
huggingface-hub==0.8.1
response==0.5.0
tokenizers==0.10.1
transformers==4.12.5
seqeval==1.2.2
deepspeed==0.5.7
tensorboard==2.7.0
seqeval==1.2.2
sentencepiece
timm==0.4.12
Pillow
einops
textdistance
shapely
```
Issue: ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

Comment:
Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default. 

CC @mariosasko
Issue: ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

Comment:
Maybe we can add a `ignore_hidden_files` flag to the builder configs of our packaged loaders (to be consistent across all of them), wdyt @lhoestq @albertvillanova?
Issue: ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

Comment:
I think they should always ignore them actually ! Not sure if adding a flag would be helpful
Issue: ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

Comment:
@lhoestq But what if the user explicitly requests those files via regex?

`glob.glob` ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's `glob` doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?
Issue: ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

Comment:
> @lhoestq But what if the user explicitly requests those files via regex?

Usually hidden files are meant to be ignored. If they are data files, they must be placed outside a hidden directory in the first place right ? I think it's more sensible to explain this than adding a flag.

> glob.glob ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's glob doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

After globbing using `fsspec`, we already ignore files that start with a `.` in `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository`, I guess we can just account for parent directories as well ?

We could open an issue on `fsspec` but I think they won't change this since it's an important breaking change for them.
Issue: Allow downloading just some columns of a dataset

Comment:
In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess