SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

Comment:
Im facing the same issue. I did run the upgrade command but that doesnt seem to resolve the issue
Issue: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

Comment:
Hi @aneeshjain, could you please specify which `huggingface_hub` version you are using?

Besides that, please run `datasets-cli env` and copy-and-paste its output below.
Issue: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

Comment:
The problem seems to be with the latest version of `datasets`. After running `pip install -U datasets huggingface_hub`, I get the following: 

```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/opt/conda/lib/python3.6/site-packages/datasets/data_files.py", line 122, in <module>
    allowed_extensions: Optional[list] = None,
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
````
Note that pip reports the latest `datasets` version as 
```bash
 pip show datasets
Name: datasets
Version: 1.14.0
```
However, if I downgrade datasets with `pip install datasets==1.11.0`, things now work
```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
dvers=1.11.0
````
Issue: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

Comment:
> Hi @JTWang2000, thanks for reporting.
> 
> However, I cannot reproduce your reported bug:
> 
> ```python
> >>> from datasets import load_dataset
> 
> >>> dataset = load_dataset("sst", "default")
> >>> dataset
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 8544
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 1101
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 2210
>     })
> })
> ```
> 
> Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
> 
> ```
> pip install -U huggingface_hub
> ```

My problem solved after updating huggingface hub command. Thanks a lot and sorry for the late reply.
Issue: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

Comment:
@tjruwase, please note that versions of `datsets` and `huggingface_hub` must be compatible one with each other:
- In `datasets` version `1.11.0`, the requirement on `huggingface_hub` was: `huggingface_hub<0.1.0`
  https://github.com/huggingface/datasets/blob/2cc00f372a96133e701275eec4d6b26d15257289/setup.py#L90
  - Therefore, your installed `huggingface_hub` version `0.0.8` was compatible
- In `datasets` version `1.12.0`, the requirement on `huggingface_hub` was: `huggingface_hub>=0.0.14,<0.1.0`
  https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/setup.py#L104
  - Therefore, your installed `huggingface_hub` version `0.0.8` was no longer compatible 
- Currently, in `datasets` version `1.15.1`, the requirement on `huggingface_hub` is: `huggingface_hub>=0.1.0,<1.0.0`
  https://github.com/huggingface/datasets/blob/018100679d21cf27136f0eccb1c50e3a9c968ce2/setup.py#L102

@JTWang2000, thanks for your answer. I close this issue then.
Issue: `cast_column` makes audio decoding fail

Comment:
Thanks for reporting, @patrickvonplaten.

I think the issue is related to mp3 resampling, not to `cast_column`.

You can check that `cast_column` works OK with non-mp3 audio files:
```python
from datasets import load_dataset
import datasets
ds = load_dataset("arabic_speech_corpus", split="train")
ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
print(ds[0]["audio"])
```

I'm fixing it.
Issue: Support loading a dataset from SQLite files

Comment:
for reference Kaggle has a good number of open source datasets stored in sqlite

Alternatively a tutorial or tool on how to convert from sqlite to parquet would be cool too
Issue: Support loading a dataset from SQLite files

Comment:
Hello, could we leverage [`pandas.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) for this? 

This would be basically the same as [`CSVBuilder`](https://github.com/huggingface/datasets/blob/7380140accf522a4363bb56c0b77a4190f49bed6/src/datasets/packaged_modules/csv/csv.py#L127)
, but uses `pandas.read_sql(..., chunksize=1)` instead of `pandas.read_csv(..., iterator=True)`

I'm happy to work on this :) 

self-assign
Issue: Error loading json dataset with multiple splits if keys in nested dicts have a different order

Comment:
Hi, 

even Pandas, which is less strict compared to PyArrow when it comes to reading JSON, doesn't support different orderings:
```python
import io
import pandas as pd

s = """
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
"""

buffer = io.StringIO(s)
df = pd.read_json(buffer, lines=True)

print(df.shape[0]) # 0
```

So we can't even fall back to Pandas in such cases.

It seems the only option is a script that recursively re-orders fields to enforce deterministic order:
```python
with open("train.json", "r") as fin:
    with open("train_reordered.json", "w") as fout:
        for line in fin:
            obj_jsonl = json.loads(line.strip())
            fout.write(json.dumps(obj_jsonl, sort_keys=True) + "\n")
```
Issue: `blog_authorship_corpus` is broken

Comment:
Hi @fdtomasi, thanks for reporting.

You are right: the original host data URL does no longer exist.

I've contacted the authors of the dataset to ask them if they host this dataset in another URL.
Issue: `blog_authorship_corpus` is broken

Comment:
Hi, @fdtomasi, the URL is fixed.

The fix is already in our master branch and it will be accessible in our next release.

In the meantime, you can include the fix if you install the `datasets` library from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```
Issue: JNLPBA Dataset

Comment:
# Steps to reproduce

To reproduce:

```python
from datasets import load_dataset
dataset = load_dataset('jnlpba')

dataset['train'].features['ner_tags']
```
Output:
```python
Sequence(feature=ClassLabel(num_classes=3, names=['O', 'B', 'I'], names_file=None, id=None), length=-1, id=None)
```
Issue: VisibleDeprecationWarning when using `set_format("numpy")`

Comment:
I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)
Issue: Import error installing with ppc64le

Comment:
This seems to be an issue with importing PyArrow so I posted the problem [here](https://issues.apache.org/jira/browse/ARROW-14323), and I'm closing this issue.
Issue: Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

Comment:
Hi @zixiliuUSC, 

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```
Issue: Make `interleave_datasets` more robust

Comment:
Hi ! Sorry for the late response

I agree `interleave_datasets` would benefit a lot from having more flexibility. If I understand correctly it would be nice to be able to define stopping strategies like `stop="first_exhausted"` (default) or `stop="all_exhausted"`. If you'd like to contribute this feature I'd be happy to give you some pointers :)

Also one can already set the max number of iterations per dataset by doing `dataset.take(n)` on the dataset that should only have `n` samples.

Regarding the `iter_cnt` counter, I think this requires a bit more thoughts, since we might have to be able to backpropagate the the counter if `map` or other transforms have been applied after `interleave_datasets`.
Issue: Make `interleave_datasets` more robust

Comment:
@sbmaruf I just notice that (1)`interleave_datasets` only samples indices once and reuse for all epochs, and (2) it's limited by the smallest dataset. Do you figure out an alternative way to achieve the same purpose?
Issue: Windows CI is unable to test streaming properly because of SSL issues

Comment:
I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```
Issue: Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?) 

Comment:
@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.
Issue: Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?) 

Comment:
Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer
Issue: load_dataset('openwebtext') yields "Compressed file ended before the end-of-stream marker was reached"

Comment:
Hi @RylanSchaeffer, thanks for reporting.

I'm sorry, but I was not able to reproduce your problem.

Normally, the reason for this type of error is that, during your download of the data files, this was not fully complete.

Could you please try to load the dataset again but forcing its redownload? Please use:
```python
dataset = load_dataset("openwebtext", download_mode="FORCE_REDOWNLOAD")
```

Let me know if the problem persists.
Issue: load_dataset('openwebtext') yields "Compressed file ended before the end-of-stream marker was reached"

Comment:
I close this issue for the moment. Feel free to re-open it again if the problem persists.
Issue: Dataset wikipedia and Bookcorpusopen cannot be fetched from dataloader.

Comment:
Hi ! I think this issue is more related to the `transformers` project. Could you open an issue on https://github.com/huggingface/transformers ?

Anyway I think the issue could be that both wikipedia and bookcorpusopen have an additional "title" column, contrary to wikitext which only has a "text" column. After calling `load_dataset`, can you try doing `dataset = dataset.remove_columns("title")` ?
Issue: Dataset wikipedia and Bookcorpusopen cannot be fetched from dataloader.

Comment:
Removing the "title" column works! Thanks for your advice.

Maybe I should still create an issue to `transformers' to mark this solution?
Issue: Error in per class precision computation 

Comment:
Hi @tidhamecha2, thanks for reporting.

Indeed, we fixed this issue just one week ago: #3008

The fix will be included in our next version release.

In the meantime, you can incorporate the fix by installing `datasets` from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```
Issue: load_dataset('the_pile_openwebtext2') produces ArrowInvalid, value too large to fit in C integer type

Comment:
I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).
Issue: load_dataset cannot download the data and hangs on forever if cache dir specified

Comment:
Issue was environment inconsistency, updating packages did the trick

`conda install -c huggingface -c conda-forge datasets`

> Collecting package metadata (current_repodata.json): done
> Solving environment: |
> The environment is inconsistent, please check the package plan carefully
> The following packages are causing the inconsistency:
> 
>   - conda-forge/noarch::datasets==1.12.1=pyhd8ed1ab_1
>   - conda-forge/win-64::multiprocess==0.70.12.2=py38h294d835_0
> done
> 
> Package Plan
> 
>   environment location: C:\xxx\anaconda3\envs\UnBias-94-1
> 
>   added / updated specs:
>     - datasets
> 
> 
> The following NEW packages will be INSTALLED:
> 
>   dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
> 
> The following packages will be UPDATED:
> 
>   ca-certificates    pkgs/main::ca-certificates-2021.9.30-~ --> conda-forge::ca-certificates-2021.10.8-h5b45459_0
>   certifi            pkgs/main::certifi-2021.5.30-py38haa9~ --> conda-forge::certifi-2021.10.8-py38haa244fe_0
> 
> The following packages will be SUPERSEDED by a higher-priority channel:
>
Issue: Non-Matching Checksum Error with crd3 dataset

Comment:
I got the same error for another dataset (`multi_woz_v22`):

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dialog_acts.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_001.json']
```
Issue: Non-Matching Checksum Error with crd3 dataset

Comment:
I'm seeing the same issue as @RylanSchaeffer:
Python 3.7.11, macOs 11.4
datasets==1.14.0

fails on:
```python
dataset = datasets.load_dataset("multi_woz_v22")
```
Issue: Identify which shard data belongs to

Comment:
Independently of this I think it raises the need to allow multiprocessing during streaming so that we get samples from multiple shards in one batch.
Issue: Inconsistent caching behaviour when using `Dataset.map()` with a `new_fingerprint` and `num_proc>1`

Comment:
Following the discussion in #3045 if would be nice to have a way to let users have a nice experience with caching even if the function is not hashable.

Currently a workaround is to make the function picklable. This can be done by implementing a callable class instead, that can be pickled using by implementing a custom `__getstate__` method for example.

However it sounds pretty complicated for a simple thing. Maybe one idea would be to have something similar to streamlit: they allow users to register the hashing of their own objects.

See the documentation about their `hash_funcs` here: https://docs.streamlit.io/library/advanced-features/caching#the-hash_funcs-parameter

Here is the example they give:

```python
class FileReference:
    def __init__(self, filename):
        self.filename = filename

def hash_file_reference(file_reference):
    filename = file_reference.filename
    return (filename, os.path.getmtime(filename))

@st.cache(hash_funcs={FileReference: hash_file_reference})
def func(file_reference):
    ...
```
Issue: Inconsistent caching behaviour when using `Dataset.map()` with a `new_fingerprint` and `num_proc>1`

Comment:
My solution was to generate a custom hash, and use the hash as a `new_fingerprint` argument to the `map()` method to enable caching. This works, but is quite hacky.

@lhoestq, this approach is very neat, this would make the whole caching mechanic more explicit. I don't have so much time to look into this right now, but I might give it a try in the future.
Issue: Inconsistent caching behaviour when using `Dataset.map()` with a `new_fingerprint` and `num_proc>1`

Comment:
Almost a year later and I'm in a similar boat. Using custom fingerprints and when using multiprocessing the cached datasets are saved with a template at the end of the filename (something like "000001_of_000008" for every process of num_proc). So if in the next time you run the script you set num_proc to a different number, the cache cannot be used.

Is there any way to get around this? I am processing a huge dataset so I do the processing on one machine and then transfer the processed data to another in its cache dir but currently that's not possible due to num_proc mismatch.
Issue: [save_to_disk] Using `select()` followed by `save_to_disk` saves complete dataset making it hard to create dummy dataset

Comment:
Hi,

the `save_to_disk` docstring explains that `flatten_indices` has to be called on a dataset before saving it to save only the shard/slice of the dataset.
Issue: [save_to_disk] Using `select()` followed by `save_to_disk` saves complete dataset making it hard to create dummy dataset

Comment:
That works! Thansk!

Might be worth doing that automatically actually in case the `save_to_disk` is called on a dataset that has an indices mapping :-)
Issue: [save_to_disk] Using `select()` followed by `save_to_disk` saves complete dataset making it hard to create dummy dataset

Comment:
I agree with @patrickvonplaten: this issue is reported recurrently, so better if we implement the `.flatten_indices()` automatically?
Issue: [save_to_disk] Using `select()` followed by `save_to_disk` saves complete dataset making it hard to create dummy dataset

Comment:
That would be great indeed - I don't really see a use case where one would not like to call `.flatten_indices()` before calling `save_to_disk`
Issue: Protect master branch to force contributions via Pull Requests

Comment:
It would be nice to protect the master from direct commits, but still having a way to merge our own PRs when no review is required (for example when updating a dataset_infos.json file, or minor bug fixes - things that happen quite often actually).
Do you know if there's a way ?
Issue: Protect master branch to force contributions via Pull Requests

Comment:
This is done. Now the master branch is protected:
- [x] Require a pull request before merging: all commits must be made to a non-protected branch and submitted via a pull request
  - Required number of approvals before merging: 1 
- [x] Require linear history: prevent merge commits from being pushed
- [x] These requirements are not enforced for administrators
- [x] Additionally, the master branch is also protected against deletion and force pushes

CC: @lhoestq @julien-c @thomwolf
Issue: `load_dataset` does not work with uploaded arrow file

Comment:
Hi ! This is not a bug, this is simply not implemented.
`save_to_disk` is for on-disk serialization and was not made compatible for the Hub.
That being said, I agree we actually should make it work with the Hub x)
Issue: Error when loading private dataset with "data_files" arg

Comment:
We'll do a release tomorrow or on wednesday to make the fix available :)

Thanks for reproting !
Issue: Resolve data_files by split name

Comment:
Awesome @lhoestq I like the proposal and it works great on my JSON community dataset. Here is the [log](https://gist.github.com/vblagoje/714babc325bcbdd5de579fd8e1648892).
Issue: Resolve data_files by split name

Comment:
From my discussion with @borisdayma it would be more general the files match if their paths contains the split name - not only if the filename contains the split name. For example for a dataset like this:
```
train/
└── data.csv
test/
└── data.csv
```

But IMO the default should be 
```
data/
├── train.csv
└── test.csv
```
because it allows people to have other directories if they have different subsets of their data (different configurations, not splits)
Issue: Resolve data_files by split name

Comment:
I just created a PR for this at https://github.com/huggingface/datasets/pull/3221, let me know what you think :)
Issue: Support multiple zipped CSV data files

Comment:
@lhoestq I would like to draw your attention to the proposed API by @lewtun, using `data_dir` to pass the ZIP URL.

I'm not totally convinced with this... What do you think?

Maybe we could discuss other approaches...

One brainstorming idea: what about using URL chaining with the hop operator in `data_files`?
Issue: Support multiple zipped CSV data files

Comment:
`data_dir` is currently exclusively used for manually downloaded data.

Maybe we can have an API that only uses data_files as you are suggesting, using URL chaining ?
```python
from datasets import load_dataset

url = "https://domain.org/filename.zip"
data_files = {"train": "zip://train_filename.csv::" + url, "test": "zip://test_filename.csv::" + url}
dataset = load_dataset("csv", data_files=data_files)
```

URL chaining is used by `fsspec` to get access to files in nested filesystems of any kind. Since `fsspec` is being used by `pandas`, `dask` and also extensively by `datasets` I think it would be nice to use it here too
Issue: Support multiple zipped CSV data files

Comment:
URL chaining sounds super nice to me! And it's also a nice way to leverage the same concepts we currently have in the docs around `fsspec` :)
Issue: Improve `get_dataset_infos`?

Comment:
To keeps things simple maybe we should use `load_dataset_builder` in `get_dataset_infos`.
`load_dataset_builder` instantiates a builder and runs the _infos() method in order to give you the most up-to-date infos, even if the dataset_infos.json is outdated or missing.
Issue: load_dataset_builder should error if "name" does not exist?

Comment:
Yes I think it should raise an error. Currently it looks like it instantiates a custom configuration with the name given by the user:
https://github.com/huggingface/datasets/blob/ba27ce33bf568374cf23a07669fdd875b5718bc2/src/datasets/builder.py#L391-L397
Issue: Chain filtering is leaking

Comment:
### Update:
I wrote a bit cleaner code snippet (without transforming to json) that can expose leaking.
```python
import datasets
import json

items = ['ab', 'c', 'df']

ds = datasets.Dataset.from_dict({'col': items})
print(list(ds))
# > Prints: [{'col': 'ab'}, {'col': 'c'}, {'col': 'df'}]

filtered = ds

# get all items that are starting with a character with ascii code bigger than 'a'
filtered = filtered.filter(lambda x: x['col'][0] > 'a', load_from_cache_file=False)
print(list(filtered))
# > Prints: [{'col': 'c'}, {'col': 'df'}] as expected

# get all items that are shorter than 2
filtered = filtered.filter(lambda x: len(x['col']) < 2, load_from_cache_file=False)
print(list(filtered))
# > Prints: [{'col': 'ab'}] -> this is a leaked item from the first filter
# > Should be: [{'col': 'c'}]
```
Issue: Chain filtering is leaking

Comment:
I just pushed a fix ! We'll do a new release soon.
In the meantime feel free to install `datasets` from source to play with it
Issue: DatasetDict.filter and Dataset.filter crashes with any "fn_kwargs" argument

Comment:
Hi @DrMatters, thanks for reporting.

This issue was fixed 14 days ago: #2950.

Currently, the fix is only in the master branch and will be made available in our next library release.

In the meantime, you can incorporate the fix by installing datasets from the master branch:
```shell
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```shell
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```