SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Fatal error with pyarrow/libarrow.so

Comment:
Thanks for reporting, @catalys1.

This seems a duplicate of:
- #3310 

The source of the problem is in PyArrow:
- [ARROW-15141: [C++] Fatal error condition occurred in aws_thread_launch](https://issues.apache.org/jira/browse/ARROW-15141)
- [ARROW-17501: [C++] Fatal error condition occurred in aws_thread_launch](https://issues.apache.org/jira/browse/ARROW-17501)

The bug in their dependency is still unresolved:
- https://github.com/aws/aws-sdk-cpp/issues/1809

Apparently, the `aws-sdk-cpp` PyArrow dependency needs to be pinned at version `1.8.186` if using conda. Have you updated it after installing PyArrow?
```shell
conda list aws-sdk-cpp
```

Maybe you should try to downgrade it to that version:
```shell
conda install -c conda-forge aws-sdk-cpp=1.8.186
```
Issue: Transfer some canonical datasets under an organization namespace

Comment:
The transfer of the dummy dataset to the dummy org works as expected:
```python
In [1]: from datasets import load_dataset; ds = load_dataset("dummy_canonical_dataset", download_mode="force_redownload"); ds
Downloading builder script: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.98k/2.98k [00:00<00:00, 2.01MB/s]
Downloading and preparing dataset dummy_canonical_dataset/default (download: 411 bytes, generated: 385 bytes, post-processed: Unknown size, total: 796 bytes) to .../.cache/huggingface/datasets/dummy_canonical_dataset/default/1.0.0/100870c358637e269fee140585e61e1472d5075a9bf6f866719934c725e55fb4...
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 411/411 [00:00<00:00, 293kB/s]
Dataset dummy_canonical_dataset downloaded and prepared to .../.cache/huggingface/datasets/dummy_canonical_dataset/default/1.0.0/100870c358637e269fee140585e61e1472d5075a9bf6f866719934c725e55fb4. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 304.16it/s]
Out[1]: 
DatasetDict({
    train: Dataset({
        features: ['langs', 'ner_tags', 'tokens'],
        num_rows: 3
    })
})

In [2]: from datasets import load_dataset; ds = load_dataset("dummy-canonical-org/dummy_canonical_dataset"); ds
Downloading builder script: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.98k/2.98k [00:00<00:00, 1.57MB/s]
Downloading and preparing dataset dummy_canonical_dataset/default to .../.cache/huggingface/datasets/dummy-canonical-org___dummy_canonical_dataset/default/1.0.0/100870c358637e269fee140585e61e1472d5075a9bf6f866719934c725e55fb4...
Dataset dummy_canonical_dataset downloaded and prepared to .../.cache/huggingface/datasets/dummy-canonical-org___dummy_canonical_dataset/default/1.0.0/100870c358637e269fee140585e61e1472d5075a9bf6f866719934c725e55fb4. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 362.48it/s]
Out[2]: 
DatasetDict({
    train: Dataset({
        features: ['langs', 'ner_tags', 'tokens'],
        num_rows: 3
    })
})
```
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
Hi ! Could it be an Out of Memory issue that could have killed one of the processes ? can you check your memory ?
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
Hi! I don't think it is a memory issue. I'm monitoring the main and spawn python processes and threads with `htop` and the memory does not peak. Besides, the example I've posted above should not be that demanding in terms of memory, right? (I have 32GB of RAM).
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
Indeed it should be fine. I couldn't reproduce the error though - I ran your script on my side and it works fine. What version of pytorch are you using ?
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
I also tried on colab and it works fine 🤔 
Maybe something is wrong with your installation of pytorch ?
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
Oh actually I just saw that you're using python 3.9

This could be related to https://github.com/huggingface/datasets/issues/4113

We'll fix that as soon as we can, in the meantime you can try to use use single process, or use an older version of python maybe ?
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
I tried with python 3.7 and the issue persists. In collab, which also uses 3.7 I don't get the issue, so yes I guess is something on mu side... will post it here if I manage to fix it
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
Hi! Which version of transformers are you using? I test the code on Colab (so python 3.7) with transformers 4.23.1, torch 1.12.1 and pyarrow 9.0.0 (also 6.x), it worked without stuck.
Issue: Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

Comment:
@bio-punk `IterableDatasetDict.map` does not support multiprocessing (only `DatasetDict.map` and `Dataset.map` do), so please open a new issue as this doesn't seem to be related to the original issue.
Issue: Mismatch between tutoriel and doc

Comment:
Hi, thanks for reporting! This line should be replaced with 
```python
dataset = dataset.map(lambda examples: tokenizer(examples["text"], return_tensors="np"), batched=True)
```
for it to work (the `return_tensors` part inside the `tokenizer` call).
Issue: load_datasets("json", ...) don't read local .json.gz properly

Comment:
Hi @junwang-wish, thanks for reporting.

Unfortunately, I'm not able to reproduce the bug. Which version of `datasets` are you using? Does the problem persist if you update `datasets`?
```shell
pip install -U datasets
```
Issue: load_datasets("json", ...) don't read local .json.gz properly

Comment:
Thanks @albertvillanova I updated `datasets` from `2.5.1` to `2.5.2` and tested copying the `json.gz` to a different directory and my mind was blown:

```python
fpath = '/data/junwang/.cache/general/57b6f2314cbe0bc45dda5b78f0871df2/test.json.gz'
ds_panda = DatasetDict(
    test=Dataset.from_pandas(
        pd.read_json(fpath, lines=True)
    )
)
ds_direct = load_dataset(
    'json', data_files={
        'test': fpath
    }, features=Features(
        text_input=Value(dtype="string", id=None),
        text_output=Value(dtype="string", id=None)
    )
)
len(ds_panda['test']), len(ds_direct['test'])
```
produces 
```python
Using custom data configuration default-0e6cf24134163e8b
Found cached dataset json (/data/junwang/.cache/huggingface/datasets/json/default-0e6cf24134163e8b/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab)
(1, 0)
```
but then I ran below command to see if the same file in a different directory leads to same discrepancy
```shell
cp /data/junwang/.cache/general/57b6f2314cbe0bc45dda5b78f0871df2/test.json.gz tmp_test.json.gz
```
and so I ran
```python
fpath = 'tmp_test.json.gz'
ds_panda = DatasetDict(
    test=Dataset.from_pandas(
        pd.read_json(fpath, lines=True)
    )
)
ds_direct = load_dataset(
    'json', data_files={
        'test': fpath
    }, features=Features(
        text_input=Value(dtype="string", id=None),
        text_output=Value(dtype="string", id=None)
    )
)
len(ds_panda['test']), len(ds_direct['test'])
```
and behold, I get 
```python
Using custom data configuration default-f679b32ab0008520
Downloading and preparing dataset json/default to /data/junwang/.cache/huggingface/datasets/json/default-f679b32ab0008520/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...
Dataset json downloaded and prepared to /data/junwang/.cache/huggingface/datasets/json/default-f679b32ab0008520/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.
(1, 1)
```
They match now !

This problem happens regardless of the shell I use (VScode jupyter extension or plain old Python REPL). 

I attached the `json.gz` here for reference: [test.json.gz](https://github.com/huggingface/datasets/files/9734843/test.json.gz)
Issue: HTTPError: 404 Client Error: Not Found for url

Comment:
Hi @km5ar, thanks for reporting.

This should be fixed in the notebook:
- the filename `datasets-issues-with-hf-doc-builder.jsonl` no longer exists on the repo; instead, current filename is `datasets-issues-with-comments.jsonl`
- see: https://huggingface.co/datasets/lewtun/github-issues/tree/main

Anyway, depending on your version of `datasets`, you can now use:
```python
from datasets import load_dataset

issues_dataset = load_dataset("lewtun/github-issues")
issues_dataset
```
instead of:
```python
from huggingface_hub import hf_hub_url

data_files = hf_hub_url(
    repo_id="lewtun/github-issues",
    filename="datasets-issues-with-hf-doc-builder.jsonl",
    repo_type="dataset",
)
from datasets import load_dataset

issues_dataset = load_dataset("json", data_files=data_files, split="train")
issues_dataset
```

Output:
```python
In [25]: ds = load_dataset("lewtun/github-issues")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.5k/10.5k [00:00<00:00, 5.75MB/s]
Using custom data configuration lewtun--github-issues-cff5093ecc410ea2
Downloading and preparing dataset json/lewtun--github-issues to .../.cache/huggingface/datasets/lewtun___json/lewtun--github-issues-cff5093ecc410ea2/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.2M/12.2M [00:00<00:00, 26.5MB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.70s/it]
Extracting data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1589.96it/s]
Dataset json downloaded and prepared to .../.cache/huggingface/datasets/lewtun___json/lewtun--github-issues-cff5093ecc410ea2/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 133.95it/s]

In [26]: ds
Out[26]: 
DatasetDict({
    train: Dataset({
        features: ['url', 'repository_url', 'labels_url', 'comments_url', 'events_url', 'html_url', 'id', 'node_id', 'number', 'title', 'user', 'labels', 'state', 'locked', 'assignee', 'assignees', 'milestone', 'comments', 'created_at', 'updated_at', 'closed_at', 'author_association', 'active_lock_reason', 'pull_request', 'body', 'timeline_url', 'performed_via_github_app', 'is_pull_request'],
        num_rows: 3019
    })
})
```
Issue: HTTPError: 404 Client Error: Not Found for url

Comment:
Thanks for reporting @km5ar and thank you @albertvillanova for the quick solution! I'll post a fix on the source too
Issue: Filtering on an empty dataset returns a corrupted dataset.

Comment:
~~It seems like #5043 fix (merged recently) is the root cause of such behaviour. When we empty indices mapping (because the dataset length equals to zero), we can no longer get column item like: `ds_filter_2['sentence']` which uses
`ds_filter_1._indices.column(0)`~~

**UPDATE:**
Empty datasets are returned without going through partial function on `map` method, which will not work to get indices for `filter`: we need to run `get_indices_from_mask_function` partial function on the dataset to get output = `{"indices": []}`. But this is complicated since functions used in args, in particular `get_indices_from_mask_function`, do not support empty datasets.
We can just handle empty datasets aside on filter method.
Issue: Bug loading `sentence-transformers/parallel-sentences`

Comment:
The dataset is sadly not really compatible to be loaded with `load_dataset`. So far it is better to git clone it and to use the files directly.

A data loading script would be needed to be added to this dataset. But this was too much overhead / not really intuitive how to create it.
Issue: Bug loading `sentence-transformers/parallel-sentences`

Comment:
Since the dataset is a bunch of TSVs we should not need a dataset script I think.

By default it tries to load all the TSVs at once, which fails here because they don't all have the same columns (pd.read_csv uses the first line as header by default). But those files have no header ! So, to properly load any TSV file in this repo, one has to pass `names=[...]` for pd.read_csv to know which column names to use.

To fix this situation, we can either do
1. replace the TSVs by TSV with column names
2. OR specify the pd.read_csv kwargs as YAML in the dataset card - and `datasets` would use that by default

WDTY ?
Issue: Bug loading `sentence-transformers/parallel-sentences`

Comment:
There are more issues in the dataset.
To load OpenSubtitles I have to provide this (see `skiprows`):

```python
df_os = pd.read_csv(
    "./parallel-sentences/OpenSubtitles/OpenSubtitles-en-de-train.tsv.gz", 
    sep="\t", 
    quoting=csv.QUOTE_NONE,
    header=None,
    names=["en", "de"],
    skiprows=[540344, 9151700, 10040173, 10040199, 11314673, 11338258, 11869223, 12159297, 12251078, 12303334],
)
```
Issue: Bug loading `sentence-transformers/parallel-sentences`

Comment:
What's wrong with those lines exactly ?
Maybe passing `error_bad_lines=False` (and maybe `warn_bad_lines=True`) can be helpful
Issue: Bug loading `sentence-transformers/parallel-sentences`

Comment:
> What's wrong with those lines exactly ? 

Stuff like this: `ParserError: Error tokenizing data. C error: Expected 2 fields in line 540345, saw 3`
Issue: Bug loading `sentence-transformers/parallel-sentences`

Comment:
>  Maybe passing error_bad_lines=False (and maybe warn_bad_lines=True) can be helpful

Yes. That would hide the issue but not solve it.
Issue: Replace AssertionErrors with more meaningful errors

Comment:
Looks like the top-level `datasource` directory was removed when https://github.com/huggingface/datasets/pull/4974 was merged, so there are 3 source files to fix.
Issue: Support default config name when no builder configs

Comment:
Thank you for creating this feature request, Albert.

For context this is the datatest where Albert has been helping me to switch to on-the-fly split config https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing

and the attempt to switch on-the-fly splits was here: https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing/discussions/2/files

but which I had to revert since providing no split breaks at run time.
Issue: `_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`

Comment:
This is maybe related to python 3.10, do you think you could try on 3.8 ?

In the meantime we'll keep improving the support for 3.10. Let me add a dedicated CI
Issue: `_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`

Comment:
I did some binary search and seems like the root cause is either `multiprocess` or `dill`. python 3.10 is fine. Specifically:
- `multiprocess==0.70.12.2, dill==0.3.4`: works
- `multiprocess==0.70.12.2, dill==0.3.5.1`: doesn't work
- `multiprocess==0.70.13, dill==0.3.5.1`: doesn't work
- `multiprocess==0.70.13, dill==0.3.4`: can't test, `multiprocess==0.70.13` requires `dill>=0.3.5.1`

I will pin their versions on my end. I don't have enough knowledge of how python multiprocessing works to debug this, but ideally there could be a fix. It's also possible that I'm doing something wrong in my code, but again the `.name` of the logger that failed to pickle is `datasets.fingerprint`, which I'm not using directly.
Issue: `_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`

Comment:
I'm not 100% sure how to figure it out -- the stack trace above doesn't clearly give me a place where I can print out who owns the logger, etc. I only found out its `.name` is `datasets.fingerprint` by printing right before
```
  File ".../logging/__init__.py", line 1774, in __reduce__
    raise pickle.PicklingError('logger cannot be pickled')
```
If you have any idea on how to find it out, please let me know.
Issue: `_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`

Comment:
Ok I see, not sure why it triggers this error though, in `logging.py` the code is

https://github.com/python/cpython/blob/c9da063e32725a66495e4047b8a5ed13e72d9e8e/Lib/logging/__init__.py#L1769-L1775

and on my side it works on 3.10 with dill 0.3.5.1 and multiprocess 0.70.13
```python
>>> datasets.fingerprint.logger.__reduce__()                                                                                                         
(<function logging.getLogger(name=None)>, ('datasets.fingerprint',))
```
Could you try to run this code ?

Are you in an environment where the loggers are instantiated differently ? Can you check the source code of `logging.Logger.__reduce__` in `".../logging/__init__.py", line 1774` ?
Issue: Unable to Use Custom Dataset Locally

Comment:
Hi ! I opened a PR in your repo to fix this :)
https://huggingface.co/datasets/zpn/pubchem_selfies/discussions/7

basically you need to use `open` for streaming to work properly
Issue: Unable to Use Custom Dataset Locally

Comment:
Thank you so much for this! Naive question, is this a feature of `open` or have you all overloaded it to be able to read from a URL? Any links to code/documentation would be greatly appreciated, I'd love to learn more
Issue: Unable to Use Custom Dataset Locally

Comment:
`datasets` extends `open` in dataset scripts to work with URLs. The builtin `open` from python only works with local files.

You can find the extension here: https://github.com/huggingface/datasets/blob/6ad430ba0cdeeb601170f732d4bd977f5c04594d/src/datasets/download/streaming_download_manager.py#L435-L451

I think we can create a docs section dedicated to streaming to explain how this works
Issue: Intermittent JSON parse error when streaming the Pile

Comment:
Maybe #2838 can help. In this PR we allow to skip bad chunks of JSON data to not crash the training

Did you have warning messages before the error ?

something like this maybe ?
```
03/24/2022 02:19:46 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [1/20]
03/24/2022 02:20:01 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [2/20]
03/24/2022 02:20:09 - ERROR - datasets.packaged_modules.json.json - Failed to read file 'gzip://file-000000000007.json::https://huggingface.co/datasets/lvwerra/codeparrot-clean-train/resolve/1d740acb9d09cf7a3307553323e2c677a6535407/file-000000000007.json.gz' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Invalid value. in row 0
```
Issue: Intermittent JSON parse error when streaming the Pile

Comment:
Ah, thanks! I did get errors like that. Sad that PR wasn't merged in! 

I'm currently just downloading 200GB of the Pile locally to avoid streaming (I have space and it's faster anyway), but that's really useful! I can probably apply the dumb patch of just commenting out the bits that raise the JSON Parse Error lol, based on your code - if I continue the loop should it be fine?
Issue: Intermittent JSON parse error when streaming the Pile

Comment:
Yup you can get some inspiration from this PR. It simply ignores the bad chunks (a chunk is ~a few MBs of data).
We'll try to merge this PR soon
Issue: Audiofolder creates empty Dataset if files same level as metadata

Comment:
Hi! Unfortunately, I can't reproduce this behavior. Instead, I get `ValueError: audio at 2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav doesn't have metadata in /audio-data/metadata.csv`, which can be fixed by removing the `./` from the file name.

(Link to a Colab that tries to reproduce this behavior: https://colab.research.google.com/drive/1IhQzULYi0Van1xLrN_SddBX1JF7mLZZK?usp=sharing)
Issue: Audiofolder creates empty Dataset if files same level as metadata

Comment:
I think we can make the file name matching part more robust by replacing `file_name` with `os.path.normpath(file_name)`, to ignore "./" among other things, in these two places:
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L319
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L388
Issue: Audiofolder creates empty Dataset if files same level as metadata

Comment:
I realized what I was doing wrong.

The documentation puts the files in a subfolder.
Once I have done that, it worked.

But l agree that this should be handled better if possible.
Issue: Automatically revert to last successful commit to hub when a push_to_hub is interrupted

Comment:
Could you share the error you got please ? Maybe the full stack trace if you have it ?

Maybe `push_to_hub` be implemented as a single commit @Wauplin ? This way if it fails, the repo is still at the previous (valid) state instead of ending-up in an invalid/incimplete state.
Issue: Automatically revert to last successful commit to hub when a push_to_hub is interrupted

Comment:
> Maybe push_to_hub be implemented as a single commit ? 

I think that would definitely be the way to go. Do you know the reasons why not implementing it like this in the first place ? I guess it is because of not been able to upload all at once with `huggingface_hub` but if there was another reason, please let me know.
About pushing all at once, it seems to be a more and more requested feature. I have created this issue https://github.com/huggingface/huggingface_hub/issues/1085 recently but other discussions already happened in the past. The `moon-landing` team is working on it (cc @coyotte508). The `huggingface_hub` integration will come afterwards.

For now, maybe it's best to wait for a proper implementation instead of creating a temporary workaround :)
Issue: Automatically revert to last successful commit to hub when a push_to_hub is interrupted

Comment:
> I think that would definitely be the way to go. Do you know the reasons why not implementing it like this in the first place ? I guess it is because of not been able to upload all at once with huggingface_hub but if there was another reason, please let me know.

Ideally we would want to upload the files iteratively - and then once everything is uploaded we proceed to commit. When we implemented `push_to_hub`, using `upload_file` for each shard was the only option.

For more context: for each shard to upload we do:
1. load the arrow shard in memory
2. convert to parquet
3. upload

So to avoid OOM we need to upload the files iteratively.

> For now, maybe it's best to wait for a proper implementation instead of creating a temporary workaround :)

Let us know if we can help !