noamaanMulla-03/datasets-issues huggingface dataset:
nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema

Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```



In 0.4.0, the assertion in `concatenate_datasets ` is on the features, and not the schema.
Could you try to update `nlp` ?

Also, since 0.4.0, you can use `dset_wikipedia.cast_(dset_books.features)` to avoid the schema cast hack.

Or maybe the assertion comes from elsewhere ?

I'm using the master branch. The assertion failure comes from the underlying `pa.concat_tables()`, which is in the pyarrow package. That method does check schemas.

Since `features.type` does not contain information about nullable vs non-nullable features, the `cast_()` method won't resolve the schema mismatch. There is information in a schema which is not stored in features.

I'm doing a refactor of type inference in #363 . Both text fields should match after that

By default nullable will be set to True

It should be good now. I was able to run

```python
>>> from nlp import concatenate_datasets, load_dataset
>>>
>>> bookcorpus = load_dataset("bookcorpus", split="train")
>>> wiki = load_dataset("wikipedia", "20200501.en", split="train")
>>> wiki.remove_columns_("title")  # only keep the text
>>>
>>> assert bookcorpus.features.type == wiki.features.type
>>> bert_dataset = concatenate_datasets([bookcorpus, wiki])
```

Thanks!
`list_datasets()` is broken.

version = '0.4.0'

`list_datasets()` is broken. It results in the following error : 

```
In [3]: nlp.list_datasets()
Out[3]: ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    375                 if cls in self.type_pprinters:
    376                     # printer registered in self.type_pprinters
--> 377                     return self.type_pprinters[cls](obj, self, cycle)
    378                 else:
    379                     # deferred printer

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in inner(obj, p, cycle)
    553                 p.text(',')
    554                 p.breakable()
--> 555             p.pretty(x)
    556         if len(obj) == 1 and type(obj) is tuple:
    557             # Special case for 1-item tuples.

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    392                         if cls is not object \
    393                                 and callable(cls.__dict__.get('__repr__')):
--> 394                             return _repr_pprint(obj, self, cycle)
    395
    396             return _default_pprint(obj, self, cycle)

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    698     """A pprint that just redirects to the normal repr function."""
    699     # Find newlines and replace them with p.break_()
--> 700     output = repr(obj)
    701     lines = output.splitlines()
    702     with p.group():

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/nlp/hf_api.py in __repr__(self)
    110
    111     def __repr__(self):
--> 112         single_line_description = self.description.replace("\n", "")
    113         return f"nlp.ObjectInfo(id='{self.id}', description='{single_line_description}', files={self.siblings})"
    114

AttributeError: 'NoneType' object has no attribute 'replace'
```


Thanks for reporting !
This has been fixed in #475 and the fix will be available in the next release

What you can do instead to get the list of the datasets is call

```python
print([dataset.id for dataset in nlp.list_datasets()])
```

Thanks @lhoestq .
Added downloading to Hyperpartisan news detection

Following the discussion on Slack and #349, I've updated the hyperpartisan dataset to pull directly from Zenodo rather than manual install, which should make this dataset much more accessible. Many thanks to @johanneskiesel !

Currently doesn't pass `test_load_real_dataset` - I'm using `self.config.name` which is `default` in this test. Might be related to #474


Thank you @ghomasHudson for making our dataset available! This is great!

The test passes since #527 :)
Benchmarks

Adding some benchmarks with DVC/CML

To add a new tracked benchmark:
- create a new python benchmarking script in `./benchmarks/`. The script can use the utilities in `./benchmarks/utils.py` and should output a JSON file with results in `./benchmarks/results/`.
- add a new pipeline stage in [dvc.yaml](./dvc.yaml) with the name of your new benchmark.

That's it
[speedup] Use indices mappings instead of deepcopy for all the samples reordering methods

Use an indices mapping instead of rewriting the dataset for all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`).

Added a `flatten_indices` method which copy the dataset to a new table to remove the indices mapping with tests.

All the samples re-ordering/selection methods should be a lot faster. The downside is that iterating on very large batch of the dataset might be a little slower when we have changed the order of the samples since with in these case we use `pyarrow.Table.take` instead of `pyarrow.Table.slice`. There is no free lunch but the speed of iterating over the dataset is rarely the bottleneck.

*Backward breaking change*: the `cache_file_name` argument in all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`) is now called `indices_cache_file_name` on purpose to make it explicit to the user that this caching file is used for caching the indices mapping and not the dataset itself.


Ok I fixed `concatenate_datasets` and added tests
Feel free to merge if it's good for you @thomwolf 

Ok, adding some benchmarks for map/filters and then I'll merge

Warning from pytorch that we should maybe consider at some point @lhoestq:
```
/__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
and PyTorch does not support non-writeable tensors. This means you can write to the underlying
(supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
protect its data or make it writeable before converting it to a tensor. This type of warning will be
suppressed for the rest of this program.
(Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
532
  return torch.tensor(x, **format_kwargs)
```

> Warning from pytorch that we should maybe consider at some point @lhoestq:
> 
> ```
> /__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
> and PyTorch does not support non-writeable tensors. This means you can write to the underlying
> (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
> protect its data or make it writeable before converting it to a tensor. This type of warning will be
> suppressed for the rest of this program.
> (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
> 532
>   return torch.tensor(x, **format_kwargs)
> ```

Not sure why we have that, it's probably linked to zero copy from arrow to numpy
nlp.load_dataset is not safe for multi processes when loading from local files

Loading from local files, e.g., `dataset = nlp.load_dataset('csv', data_files=['file_1.csv', 'file_2.csv'])`
concurrently from multiple processes, will raise `FileExistsError` from builder's line 430, https://github.com/huggingface/nlp/blob/6655008c738cb613c522deb3bd18e35a67b2a7e5/src/nlp/builder.py#L423-L438

Likely because multiple processes step into download_and_prepare, https://github.com/huggingface/nlp/blob/6655008c738cb613c522deb3bd18e35a67b2a7e5/src/nlp/load.py#L550-L554

This can happen when launching distributed training with commands like `python -m torch.distributed.launch --nproc_per_node 4` on a new collection of files never loaded before.

I can create a PR that puts in some file locks. It would be helpful if I can be informed of the convention for naming and placement of the lock.


I'll take a look!
[logging] Add centralized logging - Bump-up cache loads to warnings

Add a `nlp.logging` module to set the global logging level easily. The verbosity level also controls the tqdm bars (disabled when set higher than INFO).

You can use:
```
nlp.logging.set_verbosity(verbosity: int)
nlp.logging.set_verbosity_info()
nlp.logging.set_verbosity_warning()
nlp.logging.set_verbosity_debug()
nlp.logging.set_verbosity_error()
nlp.logging.get_verbosity() -> int
```
And use the levels:
```
nlp.logging.CRITICAL
nlp.logging.DEBUG
nlp.logging.ERROR
nlp.logging.FATAL
nlp.logging.INFO
nlp.logging.NOTSET
nlp.logging.WARN
nlp.logging.WARNING
```
Fingerprint

This PR is a continuation of #513 , in which many in-place functions were introduced or updated (cast_, flatten_) etc.
However the caching didn't handle these changes. Indeed the caching took into account only the previous cache file name of the table, and not the possible in-place transforms of the table.

To fix that, I added the concept of dataset fingerprint, that is updated after each transform (in place or not), and stored inside the table metadata.

When a dataset is created, an initial fingerprint is computed. If the dataset is memory-mapped, then the fingerprint generator doesn't read the table and only looks at the filename. However if the table is in-memory, then the fingerprint generator reads the content of the table using a batched non-crypto hashing.

I added a utility class to compute hashes of arbitrary python objects in `fingerprint.py` : `Hasher`. The API is close to standard hashing tools (`.update`, `.hexdigest`). It also supports custom hashing functions depending on object types using a registry like pickle. I added a custom hashing function to hash a `pa.Table` in a batched way, and also for `nlp.DatasetInfo` to leverage its json serialization feature.

Note about this PR:
This is a draft PR because #513 needs to be merged first.
The diff that is shown is for branches fingerprint -> indices (and not master, for now)


I changed the way I implemented fingerprint updates to use decorator functions.

I also added a new attribute called `_inplace_history` that stores the in-place history of transforms (like cast_, rename_columns, etc.). This history is useful to replay the changes that were done in-place when unpickling a dataset that is memory mapped from a file.

Let me know what you think @thomwolf
TypeError: Receiver() takes no arguments

I am trying to load a wikipedia data set

```
import nlp
from nlp import load_dataset

dataset = load_dataset("wikipedia", "20200501.en", split="train", cache_dir=data_path, beam_runner='DirectRunner')
#dataset = load_dataset('wikipedia', '20200501.sv', cache_dir=data_path, beam_runner='DirectRunner')
```

This fails in the apache beam runner. 

```
Traceback (most recent call last):
  File "D:/ML/wikiembedding/gpt2_sv.py", line 36, in <module>
    dataset = load_dataset("wikipedia", "20200501.en", split="train", cache_dir=my_cache_dir, beam_runner='DirectRunner')
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\load.py", line 548, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\builder.py", line 462, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\builder.py", line 969, in _download_and_prepare
    pipeline_results = pipeline.run()
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\pipeline.py", line 534, in run
    return self.runner.run_pipeline(self, self._options)
....
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 218, in process_encoded
    self.output(decoded_value)
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\runners\worker\operations.py", line 332, in output
    cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\Cython\Shadow.py", line 167, in cast
    return type(*args)
TypeError: Receiver() takes no arguments

```

This is run on a Windows 10 machine with python 3.8. I get the same error loading the swedish wikipedia dump.


Which version of Apache Beam do you have (can you copy your full environment info here)?

apache-beam==2.23.0
nlp==0.4.0

For me this was resolved by running the same python script on Linux (or really WSL). 

Do you manage to run a dummy beam pipeline with python on windows ? 
You can test a dummy pipeline with [this code](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py)

If you get the same error, it means that the issue comes from apache beam.
Otherwise we'll investigate what went wrong here

Still, same error, so I guess it is on apache beam then. 
Thanks for the investigation.

Thanks for trying
Let us know if you find clues of what caused this issue, or if you find a fix
No module named 'nlp.logging'

Hi, I am using nlp version 0.4.0. Trying to use bleurt as an eval metric, however, the bleurt script imports nlp.logging which creates the following error. What am I missing?

```
>>> import nlp
2020-09-02 13:47:09.210310: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> bleurt = nlp.load_metric("bleurt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/load.py", line 443, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/load.py", line 61, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/metrics/bleurt/43448cf2959ea81d3ae0e71c5c8ee31dc15eed9932f197f5f50673cbcecff2b5/bleurt.py", line 20, in <module>
    from nlp.logging import get_logger
ModuleNotFoundError: No module named 'nlp.logging'
```

Just to show once again that I can't import the logging module:

```
>>> import nlp
2020-09-02 13:48:38.190621: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> nlp.__version__
'0.4.0'
>>> from nlp.logging import get_logger
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'nlp.logging'
```


Thanks for reporting.

Apparently this is a versioning issue: the lib downloaded the `bleurt` script from the master branch where we did this change recently. We'll fix that in a new release this week or early next week. Cc @thomwolf 

Until that, I'd suggest you to download the right bleurt folder from github ([this one](https://github.com/huggingface/nlp/tree/0.4.0/metrics/bleurt)) and do

```python
from nlp import load_metric

bleurt = load_metric("path/to/bleurt/folder")
```

To download it you can either clone the repo or download the `bleurt.py` file and place it in a folder named `bleurt` 

Actually we can fix this on our side, this script didn't had to be updated. I'll do it in a few minutes
[BUGFIX] Solving mismatched checksum issue for the LinCE dataset (#539)

Hi,

I have added the updated `dataset_infos.json` file for the LinCE benchmark. This update is to fix the mismatched checksum bug #539 for one of the datasets in the LinCE benchmark. To update the file, I run this command from the nlp root directory:

```
python nlp-cli test ./datasets/lince --save_infos --all_configs
```

**NOTE**: I needed to  change [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/commands/dummy_data.py#L8) from: `from .utils.logging import get_logger` to `from nlp.utils.logging import get_logger`, otherwise the script was not able to import `get_logger`. However, I did not include that in this PR since that could have been just my environment (and another PR could be fixing this already if it is actually an issue).


Thanks a lot for that!
The line you are mentioning is a bug indeed, do you mind fixing it at the same time?

No worries! 

I pushed right away the fix, but then I realized that the master branch already had it, so I ended up merging the master branch with lince locally and then overwriting the previous commit in origin/lince. Hopefully, this is not too messy :)
Webis tl-dr

Add the Webid TL:DR dataset.


Should that rather be in an organization scope, @thomwolf @patrickvonplaten ?

> Should that rather be in an organization scope, @thomwolf @patrickvonplaten ?

I'm a bit indifferent - both would be fine for me!

@jplu - if creating the dummy_data is too tedious, I can do it as well :-) 

There is dummy_data here, no ?

Yeah I think naming it webis/tl_dr would be best @jplu if that works for you

No problem at all!! On it^^

> There is dummy_data here, no ?

Some paths were wrong - the structure is really confusing and the error messages don't really help either - I have to think about how to make this easier to understand!

Hope it was ok that I fiddled with your PR !

> Some paths were wrong - the structure is really confusing and the error message don't really help either - I have to think about how to make this easier to understand!

Oh ok! I haven't noticed that sorry :(

> Hope it was ok that I fiddled with your PR !

Of course it was ok :)

@julien-c Looks like what you have in mind?

```python
import nlp
nlp.load_dataset("datasets/webis", "tl_dr")

#Output: Downloading and preparing dataset webis/tl_dr (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/jplu/.cache/huggingface/datasets/webis/tl_dr/1.0.0...
```

Merging this for now. Maybe we can see whether to rename it in a different PR @julien-c ? 


Hi, 
Author here of the webis-tldr corpus. Any plans on integrating this dataset into the hub? I remember we could access it in the previous versions of the library. If there is a particular issue that I can help with, do let me know.

Thanks!

Hi @shahbazsyed, this dataset _is_ inside the hub but it's namespaced by the organization name `webis`.

You can load it following the steps described in https://huggingface.co/datasets/webis/tl_dr

Here's a Colab showcasing that it works: https://colab.research.google.com/drive/11IrzRVpnMLJZ8_UFFHLR8FhiajjAHRUU?usp=sharing

The reason the code is in S3 and not in this repo is that the dataset is namespaced under the `webis` organization. We don't have a lot of namespaced datasets yet but this should become the main way we add more datasets in the future.
Let us know if that's an issue for you. Thank you!
Remove logger pickling to fix gg colab issues

A `logger` objects are not picklable in google colab, contrary to `logger` objects in jupyter notebooks or in python shells.
It creates some issues in google colab right now.

Indeed by calling any `Dataset` method, the fingerprint update pickles the transform function, and as the logger comes with it, it results in an error (full stacktrace [here](http://pastebin.fr/64330)):

```python
/usr/local/lib/python3.6/dist-packages/zmq/backend/cython/socket.cpython-36m-x86_64-linux-gnu.so in zmq.backend.cython.socket.Socket.__reduce_cython__()

TypeError: no default __reduce__ due to non-trivial __cinit__
```

To fix that I no longer dump the transform (`_map_single`, `select`, etc.), but the full name only (`nlp.arrow_dataset.Dataset._map_single`, `nlp.arrow_dataset.Dataset.select`, etc.)