Pickling error when loading dataset
State: closed | Author: kandorm (NONE)

Hi,

I modified line 136 in the original [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) as:

```
# line 136: return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=args.block_size), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
return dataset
```

When I run this with transformers (3.1.0) and nlp (0.4.0), I get the following error:

```
Traceback (most recent call last):
  File "src/run_language_modeling.py", line 319, in <module>
    main()
  File "src/run_language_modeling.py", line 248, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "src/run_language_modeling.py", line 139, in get_dataset
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True, truncation=True, max_length=args.block_size), batched=True)
  File "/data/nlp/src/nlp/arrow_dataset.py", line 1136, in map
    new_fingerprint=new_fingerprint,
  File "/data/nlp/src/nlp/fingerprint.py", line 158, in wrapper
    self._fingerprint, transform, kwargs_for_fingerprint
  File "/data/nlp/src/nlp/fingerprint.py", line 105, in update_fingerprint
    hasher.update(transform_args[key])
  File "/data/nlp/src/nlp/fingerprint.py", line 57, in update
    self.m.update(self.hash(value).encode("utf-8"))
  File "/data/nlp/src/nlp/fingerprint.py", line 53, in hash
    return cls.hash_default(value)
  File "/data/nlp/src/nlp/fingerprint.py", line 46, in hash_default
    return cls.hash_bytes(dumps(value))
  File "/data/nlp/src/nlp/utils/py_utils.py", line 362, in dumps
    dump(obj, file)
  File "/data/nlp/src/nlp/utils/py_utils.py", line 339, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 446, in dump
    StockPickler.dump(self, obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 1438, in save_function
    obj.__dict__, fkwdefaults), obj=obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 1170, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 605, in save_reduce
    save(cls)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 1365, in save_type
    obj.__bases__, _dict), obj=obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 933, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 933, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 507, in save
    self.save_global(obj, rv)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 927, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle typing.Union[str, NoneType]: it's not the same object as typing.Union
```

Comments: When I change from python3.6 to python3.8, it works! 

Does it work when you install `nlp` from source on python 3.6?

No, still the pickling error.

I wasn't able to reproduce on google colab (python 3.6.9 as well) with 

pickle==4.0
dill=0.3.2
transformers==3.1.0
datasets=1.0.1 (also tried nlp 0.4.0)

If I try

```python
from datasets import load_dataset  # or from nlp
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=512), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
```
It runs without error

Closing since it looks like it's working on >= 3.6.9
Feel free to re-open if you have other questions :)