Cannot upload my own dataset

Cannot upload my own dataset
I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
```
New commands are
```
nlp-cli upload_dataset <path/to/dataset>
nlp-cli upload_metric <path/to/metric>
nlp-cli s3_datasets {rm, ls}
nlp-cli s3_metrics {rm, ls}
```
Closing this issue.


[Feature request] separate split name and split instructions
Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ?
If your dataset is a collection of sub-datasets, you should probably consider having one config per sub-dataset. For example for Glue, we have sst2, mnli etc.
If you want to have multiple train sets (for example one per stage). The easiest solution would be to name them `nlp.Split("train_stage1")`, `nlp.Split("train_stage2")`, etc. or something like that.


[Feature request] separate split name and split instructions
Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ?
Thanks for the tip! I ended up setting up three different versions of the dataset with their own configs.

for the named splits, I was trying with `nlp.Split("train-stage1")`, which fails. Changing to `nlp.Split("train_stage1")` works :) I looked for examples of what works in the code comments, it may be worth adding some examples of valid/invalid names in there?


Clone not working on Windows environment
Cloning in a windows environment is not working because of use of special character '?' in folder name ..
Please consider changing the folder name ....
Reference to folder -
nlp/datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs/dailymail/stories/

error log:
fatal: cannot create directory at 'datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs': Invalid argument

Should be fixed on master now :)


Clone not working on Windows environment
Cloning in a windows environment is not working because of use of special character '?' in folder name ..
Please consider changing the folder name ....
Reference to folder -
nlp/datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs/dailymail/stories/

error log:
fatal: cannot create directory at 'datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs': Invalid argument

Thanks @lhoestq 👍  Now I can uninstall WSL and get back to work with windows.🙂


Loading 'wikitext' dataset fails

Loading 'wikitext' dataset fails
Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
Hi, make sure you have a recent version of pyarrow.

Are you using it in Google Colab? In this case, this error is probably the same as #128


Loading 'wikitext' dataset fails

Loading 'wikitext' dataset fails
Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
Thanks!

Yes I'm using Google Colab, it seems like a duplicate then.


Loading 'wikitext' dataset fails

Loading 'wikitext' dataset fails
Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
Closing as it is a duplicate


Loading 'wikitext' dataset fails

Loading 'wikitext' dataset fails
Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
Hi,
The squad bug seems to be fixed, but the loading of the 'wikitext' still suffers from this problem (on Colab with pyarrow=0.17.1).


Loading 'wikitext' dataset fails

Loading 'wikitext' dataset fails
Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.


Loading 'wikitext' dataset fails

Loading 'wikitext' dataset fails
Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
That was it, thanks!


Add a method to shuffle a dataset

Add a method to shuffle a dataset
Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?
+1 for the naming convention

About the `shuffle` method, from my understanding it should be done in `Dataloader` (better separation between dataset processing - usage)


Add a method to shuffle a dataset

Add a method to shuffle a dataset
Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?
+1 for shuffle in `Dataloader`.
Some `Dataloader` just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.


Add a method to shuffle a dataset

Add a method to shuffle a dataset
Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?
As you might already know the issue of dataset shuffling came up in the nlp code [walkthrough](https://youtu.be/G3pOvrKkFuk?t=3204) by Yannic Kilcher


Add a method to shuffle a dataset

Add a method to shuffle a dataset
Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?
We added the `.shuffle` method :)

Closing this one.


Add Spanish POR and NER Datasets

Add Spanish POR and NER Datasets
Hi guys,
In order to cover multilingual support a little step could be adding standard Datasets used for Spanish NER and POS tasks.
I can provide it in raw and preprocessed formats.
Hello @mrm8488, are these datasets official datasets published in an NLP/CL/ML venue?


Add Spanish POR and NER Datasets

Add Spanish POR and NER Datasets
Hi guys,
In order to cover multilingual support a little step could be adding standard Datasets used for Spanish NER and POS tasks.
I can provide it in raw and preprocessed formats.
What about this one: https://github.com/ccasimiro88/TranslateAlignRetrieve?


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
Sounds good, @mariamabarham do you want to give a look?
I think we should have two configurations so we can allow either version of the dataset to be loaded with the `1.0` version being the default maybe.

Cc some authors of the great cos-e: @nazneenrajani @bmccann


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
cos_e v1.0 is related to CQA v1.0 but only CQA v1.11 dataset is available on their website. Indeed their is lots of ids in cos_e v1, which are  not in CQA v1.11  or the other way around.
@sarahwie, @thomwolf, @nazneenrajani,  @bmccann  do you know where I can  find CQA v1.0


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
@mariamabarham I'm also not sure where to find CQA 1.0. Perhaps it's not possible to include this version of the dataset. I'll close the issue if that's the case.


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
I do have a copy of the dataset. I can upload it to our repo.


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
Great @nazneenrajani. let me know once done.
Thanks


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
@mariamabarham @sarahwie I added them to the cos-e repo https://github.com/salesforce/cos-e/tree/master/data/v1.0


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
You can now do
```python
from nlp import load_dataset
cos_e = load_dataset("cos_e", "v1.0")
```
Thanks @mariamabarham !


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
Thanks!


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
@mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).


[Feature request] Add cos-e v1.0

[Feature request] Add cos-e v1.0
I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
> @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).

In the new version of `nlp`, if you try `cos_e = load_dataset("cos_e")` it throws this error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['v1.0', 'v1.11']
Example of usage:
	`load_dataset('cos_e', 'v1.0')`
```
For datasets with at least two configurations, we now force the user to pick one (no default)


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
usually you can replace `download` in your dataset script with `download_and_prepare()` - could you share the code for your dataset here? :-)


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
I have an initial version here: https://github.com/EntilZha/nlp/tree/master/datasets/qanta Thats pretty close to what I'll do as a PR, but still want to do some more sanity checks/tests (just got tests passing).

I figured out how to get all tests passing by adding a download command and some finagling with the data zip https://github.com/EntilZha/nlp/blob/master/tests/utils.py#L127


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
I'm quite positive that you can just replace the `dl_manager.download()` statements here: https://github.com/EntilZha/nlp/blob/4d46443b65f1f756921db8275594e6af008a1de7/datasets/qanta/qanta.py#L194 with `dl_manager.download_and_extract()` even though you don't extract anything. I would prefer to avoid adding more functions to the MockDataLoadManager and keep it as simple as possible (It's already to complex now IMO).

Could you check if you can replace the `download()` function?


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
I might be doing something wrong, but swapping those two gives this error:
```
>       with open(path) as f:
E       IsADirectoryError: [Errno 21] Is a directory: 'datasets/qanta/dummy/mode=first,char_skip=25/2018.4.18/dummy_data-zip-extracted/dummy_data'

src/nlp/datasets/qanta/3d965403133687b819905ead4b69af7bcee365865279b2f797c79f809b4490c3/qanta.py:280: IsADirectoryError

During handling of the above exception, another exception occurred:
```

So it seems like the directory name is getting passed. Is this not functioning as expected, or is there some caching happening maybe? I deleted the dummy files and re-ran the import script with no changes. I'm digging a bit in with a debugger, but no clear reason yet


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
From what I can tell here: https://github.com/huggingface/nlp/blob/master/tests/utils.py#L115

1. `data_url` is the correct http link
2. `path_to_dummy_data` is a directory, which is causing the issue

That path comes from `download_dummy_data`, which I think assumes that the data comes from the zip file, but isn't aware of individual files. So it seems like it data manager needs to be aware if the url its getting is for a file or a zip/directory, and pass this information along. This might happen in `download_dummy_data`, but probably better to happen in `download_and_extract`? Maybe a simple check to see if `os.path.basename` returns the dummy data zip filename, if not then join paths with the basename of the url?


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
I think the dataset script works correctly. Just the dummy data structure seems to be wrong. I will soon add more commands that should make the create of the dummy data easier.

I'd recommend that you won't concentrate too much on the dummy data.
If you manage to load the dataset correctly via:

```python
# use local path to qanta
nlp.load_dataset("./datasets/qanta")
```

then feel free to open a PR and we will look into the dummy data problem together :-)

Also please make sure that the Version is in the format 1.0.0 (three numbers separated by two points) - not a date.


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
The script loading seems to work fine so I'll work on getting a PR open after a few sanity checks on the data.

On version, we currently have it versioned with YYYY.MM.DD scheme so it would be nice to not change that, but will it cause issues?


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
> The script loading seems to work fine so I'll work on getting a PR open after a few sanity checks on the data.
>
> On version, we currently have it versioned with YYYY.MM.DD scheme so it would be nice to not change that, but will it cause issues?

It would cause issues for sure for the tests....not sure if it would also cause issues otherwise.

I would prefer to keep the same version style as we have for other models. You could for example simply add version 1.0.0 and add a comment with the date you currently use for the versioning.

 What is your opinion regarding the version here @lhoestq @mariamabarham @thomwolf ?


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
Maybe use the YYYY.MM.DD as the config name ? That's what we are doing for wikipedia


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
> Maybe use the YYYY.MM.DD as the config name ? That's what we are doing for wikipedia

I'm not sure if this will work because the name should be unique and it seems that he has multiple config name in his data with the same version.
As @patrickvonplaten  suggested, I think  you can add a comment about the version in the data description.


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
Actually maybe our versioning format (inherited from tfds) is too strong for what we use it for?
We could allow any string maybe?

I see it more and more like an identifier for the user that we will back with a serious hashing/versioning system.- so we could let the user quite free on it.


Discussion on version identifier & MockDataLoaderManager for test data
Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
I'm good with either putting it in description, adding it to the config, or loosening version formatting. I mostly don't have a full conceptual grasp of what each identifier ends up meaning in the datasets code so hard to evaluate the best approach.

For background, the multiple formats is a consequence of:

1. Each example is one multi-sentence trivia question
2. For training, its better to treat each sentence as an example
3. For evaluation, should test on: (1) first sentence, (2) full question, and (3) partial questions (does the model get the question right having seen the first half)

We use the date format for version since: (1) we expect some degree of updates since new questions come in every year and (2) the timestamp itself matches the Wikipedia dump that it is dependent on (so similar to the Wikipedia dataset).

perhaps this is better discussed in https://github.com/huggingface/nlp/pull/169 or update title?


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

Hi @dpressel,

thanks for posting your issue! Can you maybe add a complete code snippet that we can copy paste to reproduce the error? For example, I'm not sure where the variable `train_set` comes from in your code and it seems like you are loading multiple datasets at once?


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

Hi, the full example was listed in the PR above, but here is the exact link:

https://github.com/dpressel/mead-baseline/blob/3c1aa3ca062cb23f303ca98ac40b6652b37ee971/api-examples/layers-classify-hf-datasets.py

The problem is coming from
```
            if cache_file_name is None:
                # we create a unique hash from the function, current dataset file and the mapping args
                cache_kwargs = {
                    "with_indices": with_indices,
                    "batched": batched,
                    "batch_size": batch_size,
                    "remove_columns": remove_columns,
                    "keep_in_memory": keep_in_memory,
                    "load_from_cache_file": load_from_cache_file,
                    "cache_file_name": cache_file_name,
                    "writer_batch_size": writer_batch_size,
                    "arrow_schema": arrow_schema,
                    "disable_nullable": disable_nullable,
                }
                cache_file_name = self._get_cache_file_path(function, cache_kwargs)
```
The cached value is always the same, but I was able to change that by just renaming the function each time which seems to fix the issue.


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

This fixed my issue (I think)

https://github.com/dpressel/mead-baseline/commit/48aa8ecde4b307bd3e7dde5fe71e43a1d4769ee1


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

> Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq

Oh, awesome!  I see the PR, Ill check it out


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

The PR should prevent the cache from losing track of the of the dataset type (based on the location of its data). Not sure about your second problem though (cache off).


caching in map causes same result to be returned for train, validation and test
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

Yes, with caching on, it seems to work without the function renaming hack, I mentioned this also in the PR. Thanks!


How can we add more datasets to nlp library?

Found it. https://github.com/huggingface/nlp/tree/master/datasets


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
You can just run:
`val = nlp.load_dataset('squad')`

if you want to have just the validation script you can also do:

`val = nlp.load_dataset('squad', split="validation")`


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
If you want to load a local dataset, make sure you include a `./` before the folder name.


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
This happens by just doing run all cells on colab ... I assumed the colab example is broken.


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
```
!pip uninstall -y -qq pyarrow
!pip uninstall -y -qq nlp
!pip install -qq git+https://github.com/huggingface/nlp.git
```


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
> Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
>
> ```
> !pip uninstall -y -qq pyarrow
> !pip uninstall -y -qq nlp
> !pip install -qq git+https://github.com/huggingface/nlp.git
> ```

Tried, having the same error.


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
Can you post a link here of your colab? I'll make a copy of it and see what's wrong


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
This should be fixed in the current version of the notebook. You can try it again


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
Also see: https://github.com/huggingface/nlp/issues/222


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
It seems like your download was corrupted :-/ Can you run the following command:

```
rm -r /root/.cache/huggingface/datasets
```

to delete the cache completely and rerun the download?


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
I tried the notebook again today and it worked without barfing. 👌


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
Jeez - don't know what happened there :D Should be fixed now!

Thanks a lot for reporting this @tomhosking !


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
Hi @patrickvonplaten!

I'm now getting the below error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-3206959998b9> in <module>
      1 import nlp
      2
----> 3 dataset = nlp.load_dataset('wmt14')
      4 print(dataset['train'][0])

~/.local/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    507     # Instantiate the dataset builder
    508     builder_instance = builder_cls(
--> 509         cache_dir=cache_dir, name=name, version=version, data_dir=data_dir, data_files=data_files, **config_kwargs,
    510     )
    511

TypeError: Can't instantiate abstract class Wmt with abstract methods _subsets
```


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
To correct this error I think you need the master branch of `nlp`. Can you try to install `nlp` with. `WMT` was not included at the beta release of the library.

Can you try:
`pip install git+https://github.com/huggingface/nlp.git`

and check again?


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
That works, thanks :)

The WMT datasets are listed in by `list_datasets()` in the beta release on pypi - it would be good to only show datasets that are actually supported by that version?


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
Usually, the idea is that a dataset can be added without releasing a new version. The problem in the case of `WMT` was that some "core" code of the library had to be changed as well.

@thomwolf @lhoestq @julien-c - How should we go about this. If we add a dataset that also requires "core" code changes, how do we handle the versioning? The moment a dataset is on AWS it will actually be listed with `list_datasets()` in all earlier versions...

Is there a way to somehow insert the `pip version` to the HfApi() and get only the datasets that were available for this version (at the date of the release of the version) @julien-c ?


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
We plan to have something like a `requirements.txt` per dataset to prevent user from loading dataset with old version of `nlp` or any other libraries. Right now the solution is just to keep `nlp` up to date when you want to load a dataset that leverages the latests features of `nlp`.

For datasets that are on AWS but that use features that are not released yet we should be able to filter those from the `list_dataset` as soon as we have the `requirements.txt` feature on (filter datasets that need a future version of `nlp`).

Shall we rename this issue to be more explicit about the problem ?
Something like `Specify the minimum version of the nlp library required for each dataset` ?


SyntaxError with WMT datasets

SyntaxError with WMT datasets
The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
Closing this one.
Feel free to re-open if you have other questions :)