SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: New Datasets: IWSLT15+, ITTB

Comment:
Hi @sshleifer, I'm trying to add IWSLT using the link you provided but the download urls are not working. Only `[en, de]` pair is working. For others language pairs it throws a `404` error.
Issue: Google Colab - load_dataset - PyArrow exception

Comment:
+1! this is the reason our tests are failing at [TextAttack](https://github.com/QData/TextAttack) 

(Though it's worth noting if we fixed the version number of pyarrow to 0.16.0 that would fix our problem too. But in this case we'll just wait for you all to update)
Issue: Google Colab - load_dataset - PyArrow exception

Comment:
Came to raise this issue, great to see other already have and it's being fixed so soon!

As an aside, since no one wrote this already, it seems like the version check only looks at the second part of the version number making sure it is >16, but pyarrow newest version is 1.0.0 so the second past is 0!
Issue: Google Colab - load_dataset - PyArrow exception

Comment:
> Indeed, we’ll make a new PyPi release next week to solve this. Cc @lhoestq

Yes definitely
Issue: ImportWarning for pyarrow 1.0.0

Comment:
This was fixed in #434 
We'll do a release later this week to include this fix.
Thanks for reporting
Issue: ImportWarning for pyarrow 1.0.0

Comment:
I dont know if the fix was made but the problem is still present : 
Instaled with pip : NLP 0.3.0 // pyarrow 1.0.0 
OS : archlinux with kernel zen 5.8.5
Issue: How to reuse functionality of a (generic) dataset?

Comment:
Hi @ArneBinder, we have a few "generic" datasets which are intended to load data files with a predefined format:
- csv: https://github.com/huggingface/nlp/tree/master/datasets/csv
- json: https://github.com/huggingface/nlp/tree/master/datasets/json
- text: https://github.com/huggingface/nlp/tree/master/datasets/text

You can find more details about this way to load datasets here in the documentation: https://huggingface.co/nlp/loading_datasets.html#from-local-files

Maybe your brat loading script could be shared in a similar fashion?
Issue: How to reuse functionality of a (generic) dataset?

Comment:
Hi! You can either copy&paste the builder script and import the builder from there or use `datasets.load_dataset_builder` inside the script and call the methods of the returned builder object.
Issue: [FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

Comment:
Yes, that would be nice. We could take a look at what tensorflow `tf.data` does under the hood for instance.
Issue: [FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

Comment:
So `tf.data.Dataset.map()` returns a `ParallelMapDataset` if `num_parallel_calls is not None` [link](https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/data/ops/dataset_ops.py#L1623).

There, `num_parallel_calls` is turned into a tensor and and fed to `gen_dataset_ops.parallel_map_dataset` where it looks like tensorflow takes over.

We could start with something simple like a thread or process pool that `imap`s over some shards.
Issue: [FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

Comment:
Multiprocessing was added in #552 . You can set the number of processes with `.map(..., num_proc=...)`. It also works for `filter`

Closing this one, but feel free to reo-open if you have other questions
Issue: [FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

Comment:
@lhoestq Great feature implemented! Do you have plans to add it to official tutorials [Processing data in a Dataset](https://huggingface.co/docs/datasets/processing.html?highlight=save#augmenting-the-dataset)? It took me sometime to find this parallel processing api.
Issue: Correct data structure for PAN-X task in XTREME dataset?

Comment:
Hi @lhoestq 
I made the proposed changes to the `xtreme.py` script. I noticed that I also need to change the schema in the `dataset_infos.json` file.  More specifically the `"features"` part of the PAN-X.LANG dataset:

```json
"features":{
   "word":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   },
   "ner_tag":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   },
   "lang":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   }
}
```
To fit the code above the fields `"word"`, `"ner_tag"`, and `"lang"` would become `"words"`, `ner_tags"` and `"langs"`. In addition the `dtype` should be changed from `"string"` to `"list"`.

 I made this changes but when trying to test this locally with `dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')` I face the issue that the `dataset_info.json` file is always overwritten by a downloaded version with the old settings, which then throws an error because the schema does not match. This makes it hard to test the changes locally. Do you have any suggestions on how to deal with that?
Issue: Correct data structure for PAN-X task in XTREME dataset?

Comment:
Hi !

You have to point to your local script.
First clone the repo and then:

```python
dataset = load_dataset("./datasets/xtreme", "PAN-X.en")
```
The "xtreme" directory contains "xtreme.py".

You also have to change the features definition in the `_info` method. You could use:

```python
features = nlp.Features({
    "words": [nlp.Value("string")],
    "ner_tags": [nlp.Value("string")],
    "langs": [nlp.Value("string")],
})
```

Hope this helps !
Let me know if you have other questions.
Issue: Correct data structure for PAN-X task in XTREME dataset?

Comment:
Thanks, I am making progress. I got a new error `NonMatchingSplitsSizesError ` (see traceback below), which I suspect is due to the fact that number of rows in the dataset changed (one row per word --> one row per sentence) as well as the number of bytes due to the slightly updated data structure. 

```python
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=1756492, num_examples=80536, dataset_name='xtreme'), 'recorded': SplitInfo(name='validation', num_bytes=1837109, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='test', num_bytes=1752572, num_examples=80326, dataset_name='xtreme'), 'recorded': SplitInfo(name='test', num_bytes=1833214, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='train', num_bytes=3496832, num_examples=160394, dataset_name='xtreme'), 'recorded': SplitInfo(name='train', num_bytes=3658428, num_examples=20000, dataset_name='xtreme')}]
```
I can fix the error by replacing the values in the `datasets_infos.json` file, which I tested for English. However, to update this for all 40 datasets manually is slightly painful. Is there a better way to update the expected values for all datasets?
Issue: Correct data structure for PAN-X task in XTREME dataset?

Comment:
One more thing about features. I mentioned

```python
features = nlp.Features({
    "words": [nlp.Value("string")],
    "ner_tags": [nlp.Value("string")],
    "langs": [nlp.Value("string")],
})
```

but it's actually not consistent with the way we write datasets. Something like this is simpler to read and more consistent with the way we define datasets:

```python
features = nlp.Features({
    "words": nlp.Sequence(nlp.Value("string")),
    "ner_tags": nlp.Sequence(nlp.Value("string")),
    "langs": nlp.Sequence(nlp.Value("string")),
})
```

Sorry about that
Issue: Addition of google drive links to dl_manager

Comment:
I think the problem is the way you wrote your urls. Try the following structure to see `https://drive.google.com/uc?export=download&id=your_file_id` . 

@lhoestq
Issue: Addition of google drive links to dl_manager

Comment:
Oh sorry, I think `_get_drive_url` is doing that. 

Have you tried to use `dl_manager.download_and_extract(_get_drive_url(_TRAIN_URL)`? it should work with google drive links.
Issue: from_dict delete?

Comment:
`from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
Right now if you want to use `from_dict` you have to install the package from the master branch
```
pip install git+https://github.com/huggingface/nlp.git
```
Issue: from_dict delete?

Comment:
> `from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
> Right now if you want to use `from_dict` you have to install the package from the master branch
> 
> ```
> pip install git+https://github.com/huggingface/nlp.git
> ```
OK, thank you.
Issue: Is there a way to download only NQ dev?

Comment:
Unfortunately it's not possible to download only the dev set of NQ.

I think we could add a way to download only the test set by adding a custom configuration to the processing script though.
Issue: Is there a way to download only NQ dev?

Comment:
Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others. 
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.  
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.
Issue: Is there a way to download only NQ dev?

Comment:
I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.
Issue: Unable to load XTREME dataset from disk

Comment:
Hi @lewtun, you have to provide the full path to the downloaded file for example `/home/lewtum/..`
Issue: Unable to load XTREME dataset from disk

Comment:
I was able to repro. Opening a PR to fix that.
Thanks for reporting this issue !
Issue: MissingBeamOptions for Wikipedia 20200501.en

Comment:
Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c
Issue: MissingBeamOptions for Wikipedia 20200501.en

Comment:
I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?
Issue: MissingBeamOptions for Wikipedia 20200501.en

Comment:
This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.
Issue: Faster Shuffling?

Comment:
I think the slowness here probably come from the fact that we are copying from and to python.

@lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?
Issue: Faster Shuffling?

Comment:
> @lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?

I just tried with `writer.write_table` with tables of 1000 elements and it's slower that the solution in #405 

On my side (select 10 000 examples):
- Original implementation: 12s
- Batched solution: 100ms
- solution using arrow tables: 350ms

I'll try with arrays and record batches to see if we can make it work.
Issue: Faster Shuffling?

Comment:
I tried using `.take` from pyarrow recordbatches but it doesn't improve the speed that much:
```python
import nlp
import numpy as np

dset = nlp.Dataset.from_file("dummy_test_select.arrow")  # dummy dataset with 100000 examples like {"a": "h"*512}
indices = np.random.randint(0, 100_000, 1000_000)
```

```python
%%time
batch_size = 10_000
writer = ArrowWriter(schema=dset.schema, path="dummy_path",
                     writer_batch_size=1000, disable_nullable=False)
for i in tqdm(range(0, len(indices), batch_size)):
    table = pa.concat_tables(dset._data.slice(int(i), 1) for i in indices[i : min(len(indices), i + batch_size)])
    batch = table.to_pydict()
    writer.write_batch(batch)
writer.finalize()
# 9.12s
```


```python
%%time
batch_size = 10_000
writer = ArrowWriter(schema=dset.schema, path="dummy_path", 
                     writer_batch_size=1000, disable_nullable=False)
for i in tqdm(range(0, len(indices), batch_size)):
    batch_indices = indices[i : min(len(indices), i + batch_size)]
    # First, extract only the indices that we need with a mask
    mask = [False] * len(dset)
    for k in batch_indices:
        mask[k] = True
    t_batch = dset._data.filter(pa.array(mask))
    # Second, build the list of indices for the filtered table, and taking care of duplicates
    rev_positions = {}
    duplicates = 0
    for i, j in enumerate(sorted(batch_indices)):
        if j in rev_positions:
            duplicates += 1
        else:
            rev_positions[j] = i - duplicates
    rev_map = [rev_positions[j] for j in batch_indices]
    # Third, use `.take` from the combined recordbatch
    t_combined = t_batch.combine_chunks()  # load in memory
    recordbatch = t_combined.to_batches()[0]
    table = pa.Table.from_arrays(
        [recordbatch[c].take(pa.array(rev_map)) for c in range(len(dset._data.column_names))],
        schema=writer.schema
    )
    writer.write_table(table)
writer.finalize()
# 3.2s
```
Issue: Faster Shuffling?

Comment:
Shuffling is now significantly faster thanks to #513 
Feel free to play with it now :)

Closing this one, but feel free to re-open if you have other questions
Issue: Faster Shuffling?

Comment:
> Shuffling is now significantly faster thanks to #513 Feel free to play with it now :)
> 
> Closing this one, but feel free to re-open if you have other questions

I have a similar issue. My code is 

```
    for batch_num in range(num_batches):
        print(f'--> {batch_num=}\n') if verbose else None
        # - Get batch
        shuffled_dataset = dataset.shuffle(buffer_size=buffer_size, seed=seed)
        raw_text_batch = shuffled_dataset.take(batch_size)
        tokenized_batch = map(raw_text_batch)
        if verbose:
            time_start = time.time()
            print(f'{raw_text_batch=}')
            print(f'{tokenized_batch=}')
            print(f'{next(iter(raw_text_batch))=}')
            print(f'{next(iter(tokenized_batch))=}')
            print(f'Time it took: {time.time() - time_start} seconds \a\n')
   ```

without the suffle it takes 4.5 secs with it takes 87.1 secs. Is this difference expected? my dataset version is:
```
(beyond_scale) brando9@ampere1:~/beyond-scale-language-data-diversity$ pip list | grep dataset
datasets                             2.14.3
```

@lhoestq thoughts?
Issue: Faster Shuffling?

Comment:
still slow even with update to `2.14.4` most recent as of this writing

```
Time it took: 4.301205635070801 seconds 

--> batch_num=0

raw_text_batch=<datasets.iterable_dataset.IterableDataset object at 0x7f1fea7c2a40>
tokenized_batch=<datasets.iterable_dataset.IterableDataset object at 0x7f1fea7c2f20>
next(iter(raw_text_batch))={'text': "No matter which style you choose, you can be sure of one thing: our quality and craftsmanship are the best in the business. It's who we are and what we believe in. And it's evident every day on the factory floor where our dedicated teams take great pride in every stitch.", 'timestamp': '2019-04-19T06:43:50Z', 'url': 'https://institchescustoms.com/katzkin.html'}
next(iter(tokenized_batch))={'input_ids': tensor([ 2949,  2300,   543,  3918,   345,  3853,    11,   345,   460,   307,
         1654,   286,   530,  1517,    25,   674,  3081,   290,  5977, 49820,
          389,   262,  1266,   287,   262,  1597,    13,   632,   338,   508,
          356,   389,   290,   644,   356,  1975,   287,    13,   843,   340,
          338, 10678,   790,  1110,   319,   262,  8860,  4314,   810,   674,
         7256,  3466,  1011,  1049, 11293,   287,   790, 24695,    13, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,
        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256]), 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0])}
Time it took: 102.99258613586426 seconds
```
Issue: Faster Shuffling?

Comment:
Shuffling leads to doing random access in many different locations on disk which is slower than reading contiguous data.

There is a super fast approximate shuffling algorithm implemented for iterable datasets though:

```python
iterable_dataset = dataset.to_iterable_dataset(num_shards=1024)
shuffled_dataset = iterable_dataset.shuffle(buffer_size=1000)
```

(the first batch might be a bit slow to get because the algorithm first fills a buffer before returning the first batch, see the [docs](https://huggingface.co/docs/datasets/v2.14.4/en/stream#shuffle) for more info)
Issue: 🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

Comment:
similar slow download speed here for nlp.load_dataset('wmt14', 'fr-en')
`
Downloading: 100%|██████████████████████████████████████████████████████████| 658M/658M [1:00:42<00:00, 181kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████| 918M/918M [1:39:38<00:00, 154kB/s]
Downloading:   2%|▉                                                       | 40.9M/2.37G [04:48<5:03:06, 128kB/s]
`
Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?
Issue: 🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

Comment:
> The code runs but the download speed is extremely slow, the same behaviour is not observed on wmt16 and wmt18

The original source for the files may provide slow download speeds.
We can probably host these files ourselves.

> When trying to download wmt17 zh-en, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Looks like the file`UNv1.0.en-zh.tar.gz` is missing, or the url changed. We need to fix that

> Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

Right now I don't think it's possible. Maybe @patrickvonplaten knows more about it
Issue: 🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

Comment:
Yeah, the download speed is sadly always extremely slow :-/. 
I will try to check out the `wmt17 zh-en` bug :-)
Issue: Conversion through to_pandas output numpy arrays for lists instead of python objects

Comment:
To convert from arrow type we have three options: to_numpy, to_pandas and to_pydict/to_pylist.

- to_numpy and to_pandas return numpy arrays instead of lists but are very fast.
- to_pydict/to_pylist can be 100x slower and become the bottleneck for reading data, but at least they return lists.

Maybe we can have to_pydict/to_pylist as the default and use to_numpy or to_pandas when the format (set by `set_format`) is 'numpy' or 'pandas'
Issue: [dataset] Structure of MLQA seems unecessary nested

Comment:
Same for the RACE dataset: https://github.com/huggingface/nlp/blob/master/datasets/race/race.py

Should we scan all the datasets to remove this pattern of un-necessary nesting?
Issue: to_pandas conversion doesn't always work

Comment:
Could you try to update pyarrow to >=0.17.0 ? It should fix the `to_pandas` bug

Also I'm not sure that structures like list<struct> are fully supported in the lib (none of the datasets use that).
It can cause issues when using dataset transforms like `filter` for example
Issue: TypeError when computing bertscore

Comment:
I am not able to reproduce this issue on my side.
Could you give us more details about the inputs you used ?

I do get another error though:
```
~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/bert_score/utils.py in bert_cos_score_idf(model, refs, hyps, tokenizer, idf_dict, verbose, batch_size, device, all_layers)
    371         return sorted(list(set(l)), key=lambda x: len(x.split(" ")))
    372 
--> 373     sentences = dedup_and_sort(refs + hyps)
    374     embs = []
    375     iter_range = range(0, len(sentences), batch_size)

ValueError: operands could not be broadcast together with shapes (0,) (2,)
```
That's because it gets numpy arrays as input and not lists. See #387
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
I've seen this sort of thing before -- it might help to delete the directory -- I've also noticed that there is an error with the json Dataloader for any data I've tried to load. I've replaced it with this, which skips over the data feature population step:


```python
import os

import pyarrow.json as paj

import nlp as hf_nlp

from nlp import DatasetInfo, BuilderConfig, SplitGenerator, Split, utils
from nlp.arrow_writer import ArrowWriter


class JSONDatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table

```