SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
On master, I get:

```python
>>> import datasets as ds
>>> iterable_dataset = ds.load_dataset('counter', split="train", streaming=True)
>>> rows = list(iterable_dataset.take(100))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 273, in __iter__
    yield from islice(self.ex_iterable, self.n)
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 78, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/counter/b9e4378dbd3f5ce235d2302e48168c00196e67bbcd13cc7e1f6e69ef82c0cf2a/counter.py", line 153, in _generate_examples
    files = sorted(base_path.glob(r"[0-9][0-9][0-9][0-9].xml"))
TypeError: xpathglob() missing 1 required positional argument: 'pattern'
```
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
I opened a PR to fix these issues.
Also in your test you expect a TypeError but I don't know why. On my side it works fine without raising a TypeError
Issue: Cannot download TOTTO dataset

Comment:
Hola @mrm8488, thanks for reporting.

Apparently, the data source host changed their URL one week ago: https://github.com/google-research-datasets/ToTTo/commit/cebeb430ec2a97747e704d16a9354f7d9073ff8f

I'm fixing it.
Issue: always requiring the username in the dataset name when there is one

Comment:
From what I can understand, you want the saved arrow file directory to have username as well instead of just dataset name if it was downloaded with the user prefix?
Issue: always requiring the username in the dataset name when there is one

Comment:
I don't think the user cares of how this is done, but the 2nd command should fail, IMHO, as its dataset name is invalid:
```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Moreover, if someone were to create `openwebtext-10k` w/o the prefix, they will now get the wrong dataset, if they previously downloaded `stas/openwebtext-10k`.

And if there are 2 users with the same dataset name `foo/ds` and `bar/ds` - currently this won't work to get the correct dataset.

So really there 3 unrelated issues hiding in the current behavior.
Issue: always requiring the username in the dataset name when there is one

Comment:
This has been fixed now, and we'll do a new release of the library today.

Now the stas/openwebtext-10k dataset is cached at `.cache/huggingface/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/datasets/openwebtext10k`. Since they are different, the cache won't fall back on loading the wrong one anymore.

Same for the python script used to generate the dataset: stas/openwebtext-10k is cached at `.cache/huggingface/modules/datasets_modules/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/modules/datasets_modules/datasets/openwebtext10k`
Issue: Adding GLUECoS Hinglish and Spanglish code-switching bemchmark

Comment:
Hi @yjernite I am interested in adding this dataset. 
In the repo they have also added a code mixed MT task from English to Hinglish [here](https://github.com/microsoft/GLUECoS#code-mixed-machine-translation-task). I think this could be a good dataset addition in itself and then I can add the rest of the GLUECoS tasks as one dataset. What do you think?
Issue: OpenWebText: NonMatchingSplitsSizesError

Comment:
I just regenerated the verifications metadata and noticed that nothing changed: the data file is fine (the checksum didn't change), and the number of examples is still 8013769. Not sure how you managed to get 7982430 examples.

Can you try to delete your cache ( by default at `~/.cache/huggingface/datasets`) and try again please ?
Also, on which platform are you (linux/macos/windows) ?
Issue: OpenWebText: NonMatchingSplitsSizesError

Comment:
I'll try without deleting the whole cache (we have large datasets already stored). I was under the impression that `download_mode="force_redownload"` would bypass cache.
Sorry plateform should be linux (Redhat version 8.1)
Issue: OpenWebText: NonMatchingSplitsSizesError

Comment:
Sorry I haven't had time to work on this. I'll close and re-open if I can't figure out why I'm having this issue. Thanks for taking a look !
Issue: Logging levels not taken into account

Comment:
I just take a look at all the outputs produced by `datasets` using the different log-levels.
As far as i can tell using `datasets==1.17.0` they overall issue seems to be fixed.

However, I noticed that there is one tqdm based progress indicator appearing on STDERR that I can simply not suppress.
```
Resolving data files: 100%|██████████| 652/652 [00:00<00:00, 1604.52it/s]
```

According to _get_origin_metadata_locally_or_by_urls it shold be supressable by using the `NOTSET` log-level
https://github.com/huggingface/datasets/blob/1406a04c3e911cec2680d8bc513653e0cafcaaa4/src/datasets/data_files.py#L491-L501
Sadly when specifiing the log-level `NOTSET` it seems to has no effect.

But appart from it not having any effect I must admit that it seems unintuitive to me.
I would suggest changing this such that it is only shown when the log-level is greater or equal to INFO.

This would conform better to INFO according to the [documentation](https://huggingface.co/docs/datasets/v1.0.0/package_reference/logging_methods.html#datasets.logging.set_verbosity_info).
> This will display most of the logging information and tqdm bars.

Any inputs on this?
I will be happy to supply a PR if desired 👍
Issue: Logging levels not taken into account

Comment:
Hi! This should disable the tqdm output:
```python
import datasets
datasets.set_progress_bar_enabled(False)
```

On a side note: I believe the issue with logging (not tqdm) is still relevant on master.
Issue: ArrowInvalid when mapping dataset with missing values

Comment:
Hi ! It fails because of the feature type inference.

Because the first 1000 examples all have null values in the "match" field, then it infers that the type for this field is `null` type before writing the data on disk. But as soon as it tries to map an example with a non-null "match" field, then it fails.

To fix that you can either:
- increase the writer_batch_size to >2000 (default is 1000) so that some non-null values will be in the first batch written to disk
```python
datasets = datasets.map(lambda e: {'labels': e['match']}, remove_columns=['id'], writer_batch_size=2000)
```
- OR force the feature type with:
```python
from datasets import Features, Value

features = Features({
    'conflict': Value('int64'),
    'date': Value('string'),
    'headline': Value('string'),
    'match': Value('float64'),
    'label': Value('float64')
})
datasets = datasets.map(lambda e: {'labels': e['match']}, remove_columns=['id'], features=features)
```
Issue: Add a Text Classification dataset: KanHope

Comment:
Hi ! In your script it looks like you're trying to load the dataset `bn_hate_speech,`, not KanHope.

Moreover the error `KeyError: ' '` means that you have a feature of type ClassLabel, but for a certain example of the dataset, it looks like the label is empty (it's just a string with a space). Can you make sure that the data don't have missing labels, and that your dataset script parses the labels correctly ?
Issue: The datasets.map function does not load cached dataset after moving python script

Comment:
This also happened to me on COLAB.
Details:
I ran the `run_mlm.py` in two different notebooks. 
In the first notebook, I do tokenization since I can get 4 CPU cores without any GPUs, and save the cache into a folder which I copy to drive.
In the second notebook, I copy the cache folder from drive and re-run the run_mlm.py script (this time I uncomment the trainer code which happens after the tokenization)

Note: I didn't change anything in the arguments, not even the preprocessing_num_workers
Issue: The datasets.map function does not load cached dataset after moving python script

Comment:
#2854 fixed the issue :)

We'll do a new release of `datasets` soon to make the fix available.
In the meantime, feel free to try it out by installing `datasets` from source

If you have other issues or any question, feel free to re-open the issue :)
Issue: HF_DATASETS_CACHE variable in Windows

Comment:
Agh - I'm a muppet. No quote marks are needed.
set HF_DATASETS_CACHE = C:\Datasets
works as intended.
Issue: Cannot load linnaeus dataset

Comment:
Thanks for reporting ! #2852 fixed this error

We'll do a new release of `datasets` soon :)
Issue: Downloading “reddit” dataset keeps timing out.

Comment:
It also doesn't seem to be "smart caching" and I received an error about a file not being found...
Issue: Downloading “reddit” dataset keeps timing out.

Comment:
To be clear, the error I get when I try to "re-instantiate" the download after failure is: 
```
OSError: Cannot find data file. 
Original error:
[Errno 20] Not a directory: <HOME>/.cache/huggingface/datasets/downloads/1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c/corpus-webis-tldr-17.json'
```
Issue: Downloading “reddit” dataset keeps timing out.

Comment:
Hi ! Since https://github.com/huggingface/datasets/pull/2803 we've changed the time out from 10sec to 100sec.
This should prevent the `ReadTimeoutError`. Feel free to try it out by installing `datasets` from source
```
pip install git+https://github.com/huggingface/datasets.git
```

When re-running your code you said you get a `OSError`, could you try deleting the file at the path returned by the error ? (the one after `[Errno 20] Not a directory:`). Ideally when a download fails you should be able to re-run it without error; there might be an issue here.

Finally not sure what we can do about `ConnectionError`, this must be an issue from zenodo. If it happens you simply need to try again
Issue: Downloading “reddit” dataset keeps timing out.

Comment:
@lhoestq thanks for the update. The directory specified by the OSError ie. 
```
1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c/corpus-webis-tldr-17.json 
```
 was not actually in that directory so I can't delete it.
Issue: Downloading “reddit” dataset keeps timing out.

Comment:
Oh, then could you try deleting the parent directory `1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c` instead ?
This way the download manager will know that it has to uncompress the data again
Issue: Downloading “reddit” dataset keeps timing out.

Comment:
It seems to have worked. It only took like 20min! I think the extra timeout length did the trick! One thing is that it downloaded a total of 41gb instead of 20gb but at least it finished.
Issue: cannot load data from my loacal path

Comment:
Hi ! The `data_files` parameter must be a string, a list/tuple or a python dict.

Can you check the type of your `config.train_path` please ? Or use `data_files=str(config.train_path)` ?
Issue: Remove compression from xopen

Comment:
After discussing with @lhoestq, a reasonable alternative:
- `download_manager.extract(urlpath)` adds prefixes to `urlpath` in the same way as `fsspec` does for protocols, but we implement custom prefixes for all compression formats: 
  `bz2::http://domain.org/filename.bz2`
- `xopen` parses the `urlpath` and extracts the `compression` parameter and passes it to `fsspec.open`:
  `fsspec.open("http://domain.org/filename.bz2", compression="bz2")`

Pros:
- clean solution that continues giving support to all compression formats
- no breaking change when opening non-decompressed files: if no compression-protocol-like is passed, fsspec.open does not uncompress (passes compression=None)

Cons:
- we create a "private" convention for the format of `urlpath`: although similar to `fsspec` protocols, we add custom prefixes for the `compression` argument
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
Hi @lewtun, thanks for reporting.

Apparently, `pyarrow.json` tries to cast timestamp-like fields in your JSON file to pyarrow timestamp type, and it fails with `ArrowNotImplementedError`.

I will investigate if there is a way to tell pyarrow not to try that timestamp casting.
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
I think the issue is more complex than that...

I just took one of your JSON lines and pyarrow.json read it without problem.
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
> I just took one of your JSON lines an pyarrow.json read it without problem.

yes, and for some peculiar reason the error is non-deterministic (i was eventually able to load the whole dataset by just re-running the `load_dataset` cell multiple times 🤔)

thanks for looking into this 🙏 !
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
The code works fine on my side.
Not sure what's going on here :/

I remember we did a few changes in the JSON loader in #2638 , did you do an update `datasets` when debugging this ?
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
OK after upgrading `datasets` to v1.12.1 the issue seems to have gone away. Closing this now :)
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
Oops, I spoke too soon 😓 

After deleting the cache and trying the above code snippet again I am hitting the same error. You can also reproduce it in the Colab notebook I linked to in the issue description.
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
@albertvillanova @lhoestq I noticed the same issue using datasets v1.12.1. Is there an update on when this could be fixed?
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
Apparently it's possible to make it work by increasing the `block_size`, let me open a PR
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
I just opened a PR with a fix, feel free to install `datasets` from source from source and let me know if it helps
Issue: Loading JSON throws ArrowNotImplementedError

Comment:
@zijwang did PR #3000 solve the problem for you? It did for me, so it all is good on your end we can close this issue. Thanks again to @lhoestq for the pyarrow magic 🤯
Issue: How to sample every file in a list of files making up a split in a dataset when loading?

Comment:
Hi ! This is not possible just with `load_dataset`.

You can do something like this instead:
```python
seed=42
data_files_dict = {
    "train": [train_file1, train_file2],
    "test": [test_file1, test_file2],
    "val": [val_file1, val_file2]
}
dataset = datasets.load_dataset(
    "csv",
    data_files=data_files_dict,
).shuffle(seed=seed)

sample_dataset = {splitname: split.select(range(8)) for splitname, split in dataset.items()}
```

Another alternative is loading each file separately with `split="train[:8]"` and then use `concatenate_datasets` to merge the sample of each file.
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com

Comment:
the bug code locate in ：
    if data_args.task_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset("glue", data_args.task_name, cache_dir=model_args.cache_dir)
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com

Comment:
Hi @jinec,

From time to time we get this kind of `ConnectionError` coming from the github.com website: https://raw.githubusercontent.com

Normally, it should work if you wait a little and then retry.

Could you please confirm if the problem persists?
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com

Comment:
> I can access https://raw.githubusercontent.com/huggingface/datasets/1.7.0/datasets/glue/glue.py without problem...

I can not access https://raw.githubusercontent.com/huggingface/datasets either,   I am in China
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com

Comment:
> Finally i can access it, by the superfast software. Thanks

Excuse me, I have the same problem as you, could you please tell me how to solve it？
Issue: `generate_random_fingerprint()` deterministic with 🤗Transformers' `set_seed()`

Comment:
I dug into what I believe is the root of this issue and added a repro in my comment. If this is better addressed as a cross-team issue, let me know and I can open an issue in the Transformers repo
Issue: `generate_random_fingerprint()` deterministic with 🤗Transformers' `set_seed()`

Comment:
Hi !

IMO we shouldn't try to modify `set_seed` from transformers but maybe make `datasets` have its own RNG just to generate random fingerprints.

Any opinion on this @LysandreJik ?
Issue: `ArrowInvalid: Added column's length must match table's length.` after using `select`

Comment:
Hi,

the `select` method creates an indices mapping and doesn't modify the underlying PyArrow table by default for better performance. To modify the underlying table after the `select` call, call `flatten_indices` on the dataset object as follows:
```python
from datasets import load_dataset

ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
ds = ds.flatten_indices()
ds = ds.add_column('ones', [1]*128)
```
Issue: equal operation to perform unbatch for huggingface datasets 

Comment:
Hi @lhoestq 
Maybe this is clearer to explain like this, currently map function, map one example to "one" modified one, lets assume we want to map one example to "multiple" examples, in which we do not know in advance how many examples they would be per each entry. I greatly appreciate telling me how I can handle this operation, thanks a lot
Issue: equal operation to perform unbatch for huggingface datasets 

Comment:
Hi,
this is also my question on how to perform similar operation as "unbatch" in tensorflow in great huggingface dataset library. 
thanks.
Issue: equal operation to perform unbatch for huggingface datasets 

Comment:
Hi,

`Dataset.map` in the batched mode allows you to map a single row to multiple rows. So to perform "unbatch", you can do the following:
```python
import collections

def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {ex['passage']}"
        new_batch["inputs"].extend([inputs] * len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"])
    return new_batch

dset = dset.map(unbatch, batched=True, remove_columns=dset.column_names)
```
Issue: equal operation to perform unbatch for huggingface datasets 

Comment:
Dear @mariosasko 
First, thank you very much for coming back to me on this, I appreciate it a lot. I tried this solution,  I am getting errors, do you mind
giving me one test example to be able to run your code, to understand better the format of the inputs to your function?
in this function https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L952 they copy each example to the number of "answers", do you mean one should not do the copying part and use directly your function? 


thank you very much for your help and time.
Issue: equal operation to perform unbatch for huggingface datasets 

Comment:
Hi @mariosasko 
I think finally I got this, I think you mean to do things in one step, here is the full example for completeness:

```
def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        # updates the passage.
        passage = ex['passage']
        passage = re.sub(r'(\.|\?|\!|\"|\')\n@highlight\n', r'\1 ', passage)
        passage = re.sub(r'\n@highlight\n', '. ', passage)
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {passage}"
        # duplicates the samples based on  number of answers.
        num_answers = len(ex["answers"])
        num_duplicates = np.maximum(1, num_answers)
        new_batch["inputs"].extend([inputs] * num_duplicates) #len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"] if num_answers > 0 else ["<unk>"])
    return new_batch

data = datasets.load_dataset('super_glue', 'record', split="train", script_version="master")
data = data.map(unbatch, batched=True, remove_columns=data.column_names)
```

Thanks a lot again, this was a super great way to do it.
Issue: BERTScore Error

Comment:
Hi,

The `use_fast_tokenizer` argument has been recently added to the bert-score lib. I've opened a PR with the fix. In the meantime, you can try to downgrade the version of bert-score with the following command to make the code work:
```
pip uninstall bert-score
pip install "bert-score<0.3.10"
```
Issue: English wikipedia datasets is not clean

Comment:
Hi ! Certain users might need these data (for training or simply to explore/index the dataset).

Feel free to implement a map function that gets rid of these paragraphs and process the wikipedia dataset with it before training
Issue: Add RVL-CDIP dataset

Comment:
[labels_only.tar.gz](https://docs.google.com/uc?authuser=0&id=0B0NKIRwUL9KYcXo3bV9LU0t3SGs&export=download) on the RVL-CDIP website does not work for me.

> 404. That’s an error. The requested URL was not found on this server.

I contacted the author ( Adam Harley) regarding this, and he told me that the link works for him. Not sure what the issue is. But Adam shared the file (labels_only.tar.gz) with me as an attachment.

Are we allowed to host this file(labels_only.tar.gz) elsewhere and use that link instead ?

Thank you.
Issue: Error loading C4 realnewslike dataset

Comment:
Hi @danshirron, 
`c4` was updated few days back by @lhoestq. The new configs are `['en', 'en.noclean', 'en.realnewslike', 'en.webtextlike'].` You'll need to remove any older version of this dataset you previously downloaded and then run `load_dataset` again with new configuration.