SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: [Question] Best way to batch a large dataset?

Comment:
Also note that since #322 it is now possible to do
```python
ids = [1, 10, 42, 100]
batch = dataset[ids]
```
From my experience it is quite fast but it can take lots of memory for large batches (haven't played that much with it).
Let me know if you think there could be a better way to implement it. (current code is [here](https://github.com/huggingface/nlp/blob/78628649962671b4aaa31a6b24e7275533416845/src/nlp/arrow_dataset.py#L463))
Issue: [Question] Best way to batch a large dataset?

Comment:
Thanks @lhoestq! That format is much better to work with.

I put together a benchmarking script. This doesn't measure the CPU-to-GPU efficiency, nor how it scales with multi-GPU multi-node training where many processes are making the same demands on the same dataset. But it does show some interesting results:

```python
import nlp
import numpy as np
import tensorflow as tf
import time

dset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
dset = dset.filter(lambda ex: len(ex["text"]) > 0)
bsz = 1024
n_batches = 100

def single_item_gen():
    for i in range(len(dset)):
        yield dset[i]

def sequential_batch_gen():
    for i in range(0, len(dset), bsz):
        yield dset[i:i+bsz]

def random_batch_gen():
    for i in range(len(dset)):
        indices = list(np.random.randint(len(dset), size=(bsz,)))
        yield dset[indices]

output_types = {"text": tf.string}
single_item = tf.data.Dataset.from_generator(single_item_gen, output_types=output_types).batch(bsz)
interleaved = tf.data.Dataset.range(10).interleave(
    lambda idx: tf.data.Dataset.from_generator(single_item_gen, output_types=output_types),
    cycle_length=10,
)
sequential_batch = tf.data.Dataset.from_generator(sequential_batch_gen, output_types=output_types)
random_batch = tf.data.Dataset.from_generator(random_batch_gen, output_types=output_types)

def iterate(tf_dset):
    start = time.perf_counter()
    for i, batch in enumerate(tf_dset.take(n_batches)):
        pass
    elapsed = time.perf_counter() - start
    print(f"{tf_dset} took {elapsed:.3f} secs")

iterate(single_item)
iterate(interleaved)
iterate(sequential_batch)
iterate(random_batch)
```

Results:
```
<BatchDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 23.005 secs
<InterleaveDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 0.135 secs
<FlatMapDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 0.074 secs
<FlatMapDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 0.550 secs
```

- Batching a generator which fetches a single item is terrible.
- Interleaving performs well on a single process, but doesn't scale well to multi-GPU training. I believe the bottleneck here is in Arrow dataset locking or something similar. The numbers from the table above are with interleaving.
- The sequential access dominates the random access (7x faster). Is there any way to bring random access times closer to sequential access? Maybe re-indexing the dataset after shuffling each pass over the data.
Issue: [Question] Best way to batch a large dataset?

Comment:
Hey @jarednielsen 

Thanks for this very interesting analysis!! IMHO to read text data one should use `tf.data.TextLineDataset`. It would be interesting to compare what you have done with simply load with a `TextLineDataset` and see if there is a difference.

A good example can be found here https://www.tensorflow.org/tutorials/load_data/text
Issue: [Question] Best way to batch a large dataset?

Comment:
Thanks! I'm not actually loading in raw text data, that was just the synthetic data I created for this benchmark. A more realistic use case would be a dataset of tokenized examples, which would be a dict of lists of integers. TensorFlow's TextLineDataset greedily loads the dataset into the graph itself, which can lead to out-of-memory errors - one of the main reason I'm so drawn to the `nlp` library is its zero-copy no-RAM approach to dataset loading and mapping. 

It's quite helpful for running a preprocessing pipeline - a sample ELECTRA pipeline I've built is here: https://github.com/jarednielsen/deep-learning-models/blob/nlp/models/nlp/common/preprocess.py.
Issue: [Question] Best way to batch a large dataset?

Comment:
Sorry, I think I badly expressed myself, my bad. What I suggested is to compare with the usual loading textual data in pure TF with `TextLineDataset` with `nlp`. I know it is not recommended with very large datasets to use it, but I was curious to see how it behaves compared to a processing with `nlp` on smaller datasets.

BTW your script looks very interesting, thanks for sharing!!
Issue: [Feature request] Add `shard()` method to dataset

Comment:
Hi Jared,
Interesting, thanks for raising this question. You can also do that after loading with `dataset.select()` or `dataset.filter()` which let you keep only a specific subset of rows in a dataset.
What is your use-case for sharding?
Issue: [Feature request] Add `shard()` method to dataset

Comment:
Thanks for the pointer to those functions! It's still a little more verbose since you have to manually calculate which ids each rank would keep, but definitely works.

My use case is multi-node, multi-GPU training and avoiding global batches of duplicate elements. I'm using horovod. You can shuffle indices, or set random seeds, but explicitly sharding the dataset up front is the safest and clearest way I've found to do so.
Issue: Question - Sign Language Datasets

Comment:
Even more complicating - 

As I see it, datasets can have "addons".
For example, the WebNLG dataset is a dataset for data-to-text. However, a work of mine and other works enriched this dataset with text plans / underlying text structures. In that case, I see a need to load the dataset "WebNLG" with "plans" addon.

Same for sign language - if there is a dataset of videos, one addon can be to run OpenPose, another to run ARKit4 pose estimation, and another to run PoseNet, or even just a video embedding addon. (which are expensive to run individually for everyone who wants to use these data)

This is something I dabbled with my own implementation to a [research datasets library](https://github.com/AmitMY/meta-scholar/) and I love to get the discussion going on these topics.
Issue: Question - Sign Language Datasets

Comment:
This is a really cool idea !
The example for data objects you gave for the RWTH-PHOENIX-Weather 2014 T dataset can totally fit inside the library.

For your point about formats like `ilex`, `eaf`, or `srt`, it is possible to use any library in your dataset script.
However most user probably won't need these libraries, as most datasets don't need them, and therefore it's unlikely that we will have them in the minimum requirements to use `nlp` (we want to keep it as light-weight as possible). If a user wants to load your dataset and doesn't have the libraries you need, an error is raised asking the user to install them.

More generally, we plan to have something like a `requirements.txt` per dataset. This could also be a place for addons as you said. What do you think ?
Issue: Question - Sign Language Datasets

Comment:
Thanks, Quentin, I think a `requirements.txt` per dataset will be a good thing.
I will work on adding this dataset next week, and once we sort all of the kinks, I'll add more.
Issue: Setting cache_dir gives error on wikipedia download

Comment:
Whoops didn't mean to close this one.
I did some changes, could you try to run it from the master branch ?
Issue: Error in Demo for Specific Datasets

Comment:
Thanks for reporting these errors :)

I can actually see two issues here.

First, datasets like `natural_questions` require apache_beam to be processed. Right now the import is not at the right place so we have this error message. However, even the imports are fixed, the nlp viewer doesn't actually have the resources to process NQ right now so we'll have to wait until we have a version that we've already processed on our google storage (that's what we've done for wikipedia for example).

Second, datasets like `newsroom` require manual downloads as we're not allowed to redistribute the data ourselves (if I'm not wrong). An error message should be displayed saying that we're not allowed to show the dataset.

I can fix the first issue with the imports but for the second one I think we'll have to see with @srush to show a message for datasets that require manual downloads (it can be checked whether a dataset requires manual downloads if `dataset_builder_instance.manual_download_instructions is not None`).
Issue: Error in Demo for Specific Datasets

Comment:
We don't plan to host the source files of newsroom ourselves for now.
You can still get the dataset if you follow the download instructions given by `dataset = load_dataset('newsroom')` though.
The viewer also shows the instructions now.

Closing this one. If you have other questions, feel free to re-open :)
Issue: snli -1 labels

Comment:
@jxmorris12  , we use `-1` to label examples for which `gold label`  is missing (`gold label = -` in the original dataset).
Issue: snli -1 labels

Comment:
Thanks @mariamabarham! so the original dataset is missing some labels? That is weird. Is standard practice just to discard those examples training/eval?
Issue: snli -1 labels

Comment:
Yes the original dataset is missing some labels maybe  @sleepinyourhat , @gangeli can correct me if I'm wrong 
For my personal opinion at least if you want your model to learn to predict no answer (-1) you can leave it their but otherwise you can discard them.
Issue: Cannot load arxiv dataset on MacOS?

Comment:
I couldn't replicate this issue on my macbook :/
Could you try to play with different encodings in `with open(path, encoding=...) as f` in scientific_papers.py:L108 ?
Issue: Cannot load arxiv dataset on MacOS?

Comment:
I was able to track down the file causing the problem by adding the following to `scientific_papers.py` (starting at line 116):

```python
                from json import JSONDecodeError
                try:
                    d = json.loads(line)
                    summary = "\n".join(d["abstract_text"])
                except JSONDecodeError:
                    print(path, line)
```



For me it was at: `/Users/johngiorgi/.cache/huggingface/datasets/f87fd498c5003cbe253a2af422caa1e58f87a4fd74cb3e67350c635c8903b259/arxiv-dataset/train.txt` with `"article_id": "1407.3051"`.

Not really 100% sure at the moment, but it looks like this specific substring from `"article_text"` may be causing the problem?

```
"after the missing - mass scale adjustment , the validity of the corrections was tested in the @xmath85 productions at 1.69 gev/@xmath1 . in fig .  [", "fig : calibrations ] ( a ) , we show the missing - mass spectrum in the @xmath86 region in the @xmath87 reaction at 1.69 gev/@xmath1 . a fitting result with a lorentzian function for the @xmath86  ( dashed line ) and the three - body phas
```

perhaps because it appears to be truncated. I (think) I can recreate the problem by doing the following:

```python
import json

# A minimal example of the json file that causes the error
invalid_json = '{"article_id": "1407.3051", "article_text": ["the missing - mass resolution was obtained to be 2.8 @xmath3 0.1  mev/@xmath4  ( fwhm ) , which corresponds to the missing - mass resolution of 3.2  @xmath3  0.2  mev/@xmath4  ( fwhm ) at the @xmath6 cusp region in the @xmath0 reaction .", "this resolution is at least by a factor of 2 better than the previous measurement with the same reaction ( 3.2@xmath595.5 mev/@xmath4 in @xmath84 )  @xcite .", "after the missing - mass scale adjustment , the validity of the corrections was tested in the @xmath85 productions at 1.69 gev/@xmath1 . in fig .  [", "fig : calibrations ] ( a ) , we show the missing - mass spectrum in the @xmath86 region in the @xmath87 reaction at 1.69 gev/@xmath1 . a fitting result with a lorentzian function for the @xmath86  ( dashed line ) and the three - body phas'   
# The line of code from `scientific_papers.py` which appears to cause the error
json.loads(invalid_json)
```

This is as far as I get before I am stumped.
Issue: Cannot load arxiv dataset on MacOS?

Comment:
I just checked inside `train.txt` and this line isn't truncated for me (line 163577).
Could you try to clear your cache and re-download the dataset ?
Issue: ConnectionError - Eli5 dataset download

Comment:
It should ne fixed now, thanks for reporting this one :)
It was an issue on our google storage.

Let me now if you're still facing this issue.
Issue: Error at the first example in README: AttributeError: module 'dill' has no attribute '_dill'

Comment:
It looks like the bug comes from `dill`. Which version of `dill` are you using ?
Issue: Error at the first example in README: AttributeError: module 'dill' has no attribute '_dill'

Comment:
0.2.6 is three years old now, maybe try a more recent one, e.g. the current 0.3.2 if you can?
Issue: Private/sensitive data

Comment:
Hi @MFreidank, you should already be able to load a dataset from local sources, indeed. (ping @lhoestq and @jplu)

We're also thinking about the ability to host private datasets on a hosted bucket with permission management, but that's further down the road.
Issue: Private/sensitive data

Comment:
Hi @MFreidank, it is possible to load a dataset from your local storage, but only CSV/TSV and JSON are supported. To load a dataset in JSON format:

```
nlp.load_dataset(path="json", data_files={nlp.Split.TRAIN: ["path/to/train.json"], nlp.Split.TEST: ["path/to/test.json"]})
```

For CSV/TSV datasets, you have to replace `json` by `csv`.
Issue: Private/sensitive data

Comment:
Hi @julien-c @jplu,
Thanks for sharing this solution with me, it helps, this is what I was looking for. 
If not already there and only missed by me, this could be a great addition in the docs.

Closing my issue as resolved, thanks again.
Issue: Dataset Preprocessing Cache with .map() function not working as expected

Comment:
When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?
Issue: Dataset Preprocessing Cache with .map() function not working as expected

Comment:
Thanks, that's helpful! I was running 0.1.0, but since upgraded to 0.2.1. I can't reproduce the issue anymore as I've cleared the cache & everything now seems to be running fine since the upgrade. I've added some checks to my code, so if I do encounter it again I will reopen this issue.
Issue: Dataset Preprocessing Cache with .map() function not working as expected

Comment:
Just checking in, the cache sometimes still does not work when I make changes in my processing function in version `1.2.1`. The changes made to my data processing function only propagate to the dataset when I use `load_from_cache_file=False` or clear the cache. Is this a system-specific issue?
Issue: Dataset Preprocessing Cache with .map() function not working as expected

Comment:
Hi @sarahwie 
The data are reloaded from the cache if the hash of the function you provide is the same as a computation you've done before. The hash is computed by recursively looking at the python objects of the function you provide.

If you think there's an issue, can you share the function you used or a google colab please ?
Issue: MemoryError when loading German Wikipedia

Comment:
Hi !

As you noticed, "big" datasets like Wikipedia require apache beam to be processed.
However users usually don't have an apache beam runtime available (spark, dataflow, etc.) so our goal for this library is to also make available processed versions of these datasets, so that users can just download and use them right away.

This is the case for english and french wikipedia right now: we've processed them ourselves and now they are available from our google storage. However we've not processed the german one (yet).
Issue: MemoryError when loading German Wikipedia

Comment:
Hi @lhoestq 

Thank you for your quick reply. I thought this might be the case, that the processing was done for some languages and not for others. Is there any set timeline for when other languages (German, Italian) will be processed?

Given enough memory, is it possible to process the data ourselves by specifying the `beam_runner`?
Issue: MemoryError when loading German Wikipedia

Comment:
Adding them is definitely in our short term objectives. I'll be working on this early next week :)

Although if you have an apache beam runtime feel free to specify the beam runner. You can find more info [here](https://github.com/huggingface/nlp/blob/master/docs/beam_dataset.md) on how to make it work on Dataflow but you can adapt it for Spark or any other beam runtime (by changing the `runner`).

However if you don't have a beam runtime and even if you have enough memory, I discourage you to use the `DirectRunner` on the german or italian wikipedia. According to Apache Beam documentation it was made for testing purposes and therefore it is memory-inefficient.
Issue: MemoryError when loading German Wikipedia

Comment:
I added the German and the Italian Wikipedia to our google cloud storage:
First update the `nlp` package to 0.3.0:
```bash
pip install nlp --upgrade
```
and then
```python
from nlp import load_dataset
wiki_de = load_dataset("wikipedia", "20200501.de")
wiki_it = load_dataset("wikipedia", "20200501.it")
```
The datasets are downloaded and directly ready to use (no processing).
Issue: MemoryError when loading German Wikipedia

Comment:
Hi @lhoestq 

Wow, thanks so much, that's **really** incredible! I was considering looking at creating my own Beam Dataset, as per the doc you linked, but instead opted to process the data myself using `wikiextractor`. However, now that this is available, I'll definitely switch across and use it.

Thanks so much for the incredible work, this really helps out our team considerably!

Have a great (and well-deserved ;) weekend ahead!

P.S. I'm not sure if I should close the issue here - if so I'm happy to do so.
Issue: Empty samples in glue/qqp

Comment:
We are only wrapping the original dataset.

Maybe try to ask on the GLUE mailing list or reach out to the original authors?
Issue: Empty samples in glue/qqp

Comment:
Tanks for the suggestion, I'll try to ask GLUE benchmark.
I'll first close the issue,  post the following up here afterwards, and reopen the issue if needed.
Issue: NonMatchingChecksumError when loading pubmed dataset

Comment:
For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)
Issue: PG-19

Comment:
Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?
Issue: PG-19

Comment:
Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!
Issue: [Feature request] Support for external modality for language datasets

Comment:
Thanks a lot, @aleSuglia for the very detailed and introductive feature request.
It seems like we could build something pretty useful here indeed.

One of the questions here is that Arrow doesn't have built-in support for generic "tensors" in records but there might be ways to do that in a clean way. We'll probably try to tackle this during the summer.
Issue: [Feature request] Support for external modality for language datasets

Comment:
I was looking into Facebook MMF and apparently they decided to use LMDB to store additional features associated with every example: https://github.com/facebookresearch/mmf/blob/master/mmf/datasets/databases/features_database.py
Issue: [Feature request] Support for external modality for language datasets

Comment:
I saw the Mozilla common_voice dataset in model hub, which has mp3 audio recordings as part it. It's use predominantly maybe in ASR and TTS, but dataset is a Language + Voice Dataset similar to @aleSuglia's point about Language + Vision. 

https://huggingface.co/datasets/common_voice
Issue: [Feature request] Support for external modality for language datasets

Comment:
Hey @thomwolf, are there any updates on this? I would love to contribute if possible!

Thanks, 
Alessandro
Issue: [Feature request] Support for external modality for language datasets

Comment:
Hi @aleSuglia :) In today's new release 1.17 of `datasets` we introduce a new feature type `Image` that allows to store images directly in a dataset, next to text features and labels for example. There is also an `Audio` feature type, for datasets containing audio data. For tensors there are `Array2D`, `Array3D`, etc. feature types

Note that both Image and Audio feature types take care of decoding the images/audio data if needed. The returned images are PIL images, and the audio signals are decoded as numpy arrays.

And `datasets` also leverage end-to-end zero copy from the arrow data for all of them, for maximum speed :)
Issue: Downloading dataset error with pyarrow.lib.RecordBatch

Comment:
When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.
If you don't restart, then it breaks like in your message.
Issue: documentation missing how to split a dataset

Comment:
this seems to work for my specific problem:

`self.train_ds, self.test_ds, self.val_ds = map(_prepare_ds, ('train', 'test[:25%]+test[50%:75%]', 'test[75%:]'))`
Issue: documentation missing how to split a dataset

Comment:
Currently you can indeed split a dataset using `ds_test = nlp.load_dataset('imdb, split='test[:5000]')` (works also with percentages).

However right now we don't have a way to shuffle a dataset but we are thinking about it in the discussion in #166. Feel free to share your thoughts about it.

One trick that you can do until we have a better solution is to shuffle and split the indices of your dataset:
```python
import nlp
from sklearn.model_selection import train_test_split

imdb = nlp.load_dataset('imbd', split='test')
test_indices, val_indices = train_test_split(range(len(imdb)))
```

and then to iterate each split:
```python
for i in test_indices:
    example = imdb[i]
   ...
```
Issue: documentation missing how to split a dataset

Comment:
I added a small guide [here](https://github.com/huggingface/nlp/tree/master/docs/splits.md) that explains how to split a dataset. It is very similar to the tensorflow datasets guide, as we kept the same logic.