Invalid argument with CTRLModel
Author: ChrisPalmerNZ
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (CTRLModel):

Language I am using the model on (English):

The problem arise when using:
* [X ] the official example scripts: (give details)

The tasks I am working on is:
* [X ] the Quick Tour for transformers

## To Reproduce

Steps to reproduce the behavior:

1. run the quick tour as found here https://github.com/huggingface/transformers#quick-tour
2. in the loop `for model_class, tokenizer_class, pretrained_weights in MODELS` it will crash out when processing the CTRLModel
```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-5-25349833af91> in <module>
      5 
      6     tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
----> 7     model = model_class.from_pretrained(pretrained_weights)
      8 
      9     # Encode text

g:\deeplearning\huggingface\transformers\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    389 
    390         if state_dict is None and not from_tf:
--> 391             state_dict = torch.load(resolved_archive_file, map_location='cpu')
    392 
    393         missing_keys = []

G:\Anaconda3\envs\fastai\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    384         f = f.open('rb')
    385     try:
--> 386         return _load(f, map_location, pickle_module, **pickle_load_args)
    387     finally:
    388         if new_fd:

G:\Anaconda3\envs\fastai\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    578     for key in deserialized_storage_keys:
    579         assert key in deserialized_objects
--> 580         deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
    581         if offset is not None:
    582             offset = f.tell()

OSError: [Errno 22] Invalid argument

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## What should have happened
Should have run to completion...

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version: 3.3.9
[EDIT] - type on the Python, its 3.6.9.
* PyTorch version: 1.2.0+cu92
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
> ##  Bug
> Model I am using (CTRLModel):
> 
> Language I am using the model on (English):
> 
> The problem arise when using:
> 
> * [X ] the official example scripts: (give details)
> 
> The tasks I am working on is:
> 
> * [X ] the Quick Tour for transformers
> 
> ## To Reproduce
> Steps to reproduce the behavior:
> 
> 1. run the quick tour as found here https://github.com/huggingface/transformers#quick-tour
> 2. in the loop `for model_class, tokenizer_class, pretrained_weights in MODELS` it will crash out when processing the CTRLModel
> 
> ```
> ---------------------------------------------------------------------------
> OSError                                   Traceback (most recent call last)
> <ipython-input-5-25349833af91> in <module>
>       5 
>       6     tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
> ----> 7     model = model_class.from_pretrained(pretrained_weights)
>       8 
>       9     # Encode text
> 
> g:\deeplearning\huggingface\transformers\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
>     389 
>     390         if state_dict is None and not from_tf:
> --> 391             state_dict = torch.load(resolved_archive_file, map_location='cpu')
>     392 
>     393         missing_keys = []
> 
> G:\Anaconda3\envs\fastai\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
>     384         f = f.open('rb')
>     385     try:
> --> 386         return _load(f, map_location, pickle_module, **pickle_load_args)
>     387     finally:
>     388         if new_fd:
> 
> G:\Anaconda3\envs\fastai\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
>     578     for key in deserialized_storage_keys:
>     579         assert key in deserialized_objects
> --> 580         deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
>     581         if offset is not None:
>     582             offset = f.tell()
> 
> OSError: [Errno 22] Invalid argument
> ```
> 
> ## What should have happened
> Should have run to completion...
> 
> ## Environment
> * OS:
> * Python version: 3.3.9
> * PyTorch version: 1.2.0+cu92
> * PyTorch Transformers version (or branch): 2.1.1
> * Using GPU ? Yes
> * Distributed of parallel setup ? no
> * Any other relevant information:
> 
> ## Additional context

Can you test the same code with Python version >= 3.5 and with PyTorch version 1.3.0?

Sure, l'll get back to you... but I realize that I had a typo with my Python, its 3.6.9, which I've corrected. 
Additionally, I had compiled this with `pip install -e .` from a fresh clone of the repo yesterday

OK, under Pytorch 1.3.0 it seems good - I got the following output (which I wasn't getting from 1.2.0):
```
I1111 20:26:06.461704 21676 tokenization_utils.py:374] loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json from cache at C:\Users\User\.cache\torch\transformers\a858ad854d3847b02da3aac63555142de6a05f2a26d928bb49e881970514e186.285c96a541cf6719677cfb634929022b56b76a0c9a540186ba3d8bbdf02bca42
I1111 20:26:06.464703 21676 tokenization_utils.py:374] loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt from cache at C:\Users\User\.cache\torch\transformers\aa2c569e6648690484ade28535a8157aa415f15202e84a62e82cc36ea0c20fa9.26153bf569b71aaf15ae54be4c1b9254dbeff58ca6fc3e29468c4eed078ac142
I1111 20:26:07.624670 21676 configuration_utils.py:152] loading configuration file https://storage.googleapis.com/sf-ctrl/pytorch/ctrl-config.json from cache at C:\Users\User\.cache\torch\transformers\d6492ca334c2a4e079f43df30956acf935134081b2b3844dc97457be69b623d0.1ebc47eb44e70492e0c20494a084f108332d20fea7fe5ad408ef5e7a8f2baef4
I1111 20:26:07.628657 21676 configuration_utils.py:169] Model config {
  "attn_pdrop": 0.1,
  "dff": 8192,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "from_tf": false,
  "initializer_range": 0.02,
  "is_decoder": false,
  "layer_norm_epsilon": 1e-06,
  "n_ctx": 512,
  "n_embd": 1280,
  "n_head": 16,
  "n_layer": 48,
  "n_positions": 50000,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 246534
}

I1111 20:26:08.047282 21676 modeling_utils.py:383] loading weights file https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin from cache at C:\Users\User\.cache\torch\transformers\c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
W1111 20:31:14.310047 21676 tokenization_utils.py:936] This tokenizer does not make use of special tokens. Input is returned with no modification.
W1111 20:31:14.650039 21676 tokenization_utils.py:936] This tokenizer does not make use of special tokens. Input is returned with no modification.
W1111 20:31:14.653039 21676 tokenization_utils.py:923] This tokenizer does not make use of special tokens.
```

[Here](https://colab.research.google.com/drive/1d23JwkEkS_s6XUY2_Me_MHnOrTHmNrqR#scrollTo=K4-yYInLt6YP) you can view a Google Colab written by me and **it works as expected**. I think you can close this issue.

> OK, under Pytorch 1.3.0 it seems good - I got the following output (which I wasn't getting from 1.2.0):
> 
> ```
> I1111 20:26:06.461704 21676 tokenization_utils.py:374] loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json from cache at C:\Users\User\.cache\torch\transformers\a858ad854d3847b02da3aac63555142de6a05f2a26d928bb49e881970514e186.285c96a541cf6719677cfb634929022b56b76a0c9a540186ba3d8bbdf02bca42
> I1111 20:26:06.464703 21676 tokenization_utils.py:374] loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt from cache at C:\Users\User\.cache\torch\transformers\aa2c569e6648690484ade28535a8157aa415f15202e84a62e82cc36ea0c20fa9.26153bf569b71aaf15ae54be4c1b9254dbeff58ca6fc3e29468c4eed078ac142
> I1111 20:26:07.624670 21676 configuration_utils.py:152] loading configuration file https://storage.googleapis.com/sf-ctrl/pytorch/ctrl-config.json from cache at C:\Users\User\.cache\torch\transformers\d6492ca334c2a4e079f43df30956acf935134081b2b3844dc97457be69b623d0.1ebc47eb44e70492e0c20494a084f108332d20fea7fe5ad408ef5e7a8f2baef4
> I1111 20:26:07.628657 21676 configuration_utils.py:169] Model config {
>   "attn_pdrop": 0.1,
>   "dff": 8192,
>   "embd_pdrop": 0.1,
>   "finetuning_task": null,
>   "from_tf": false,
>   "initializer_range": 0.02,
>   "is_decoder": false,
>   "layer_norm_epsilon": 1e-06,
>   "n_ctx": 512,
>   "n_embd": 1280,
>   "n_head": 16,
>   "n_layer": 48,
>   "n_positions": 50000,
>   "num_labels": 1,
>   "output_attentions": false,
>   "output_hidden_states": false,
>   "output_past": true,
>   "pruned_heads": {},
>   "resid_pdrop": 0.1,
>   "summary_activation": null,
>   "summary_first_dropout": 0.1,
>   "summary_proj_to_labels": true,
>   "summary_type": "cls_index",
>   "summary_use_proj": true,
>   "torchscript": false,
>   "use_bfloat16": false,
>   "vocab_size": 246534
> }
> 
> I1111 20:26:08.047282 21676 modeling_utils.py:383] loading weights file https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin from cache at C:\Users\User\.cache\torch\transformers\c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
> W1111 20:31:14.310047 21676 tokenization_utils.py:936] This tokenizer does not make use of special tokens. Input is returned with no modification.
> W1111 20:31:14.650039 21676 tokenization_utils.py:936] This tokenizer does not make use of special tokens. Input is returned with no modification.
> W1111 20:31:14.653039 21676 tokenization_utils.py:923] This tokenizer does not make use of special tokens.
> ```

Is there any way to get this working correctly under Pytorch 1.2? I have some work I need to continue with in that version. It doesn't matter too much as I wasn't intending to use that model, but I guess it might matter to someone else...

Thanks very much for the Colab notebook - yes it works as expected there. I am now having an issue with running the last part of the Quick Tour, but I'll post that separately, and close this one!

I'm very honest with you: I don't have the exact answer to your notable question. I'm aware of the fact that PyTorch v1.3.0 has got many changes and improvements from PyTorch v1.2, as stated [here](url). Some changes have been made on the `torch.load()` method too.

> Is there any way to get this working correctly under Pytorch 1.2? I have some work I need to continue with in that version. It doesn't matter too much as I wasn't intending to use that model, but I guess it might matter to someone else...

@ChrisPalmerNZ Would you care to re-open this issue?
There really should not be a bug with the intro sample code.

OK, re-opened - interesting to see what fixes this for Pytorch 1.2 - but I felt that @TheEdoardo93 was indicating that this will only work under Pytorch >= 1.3.0 

Hi, you seem to be having problems when loading the CTRLModel in memory. Usually problems like this can occur if behind a firewall or if the file was corrupted. Could you try to load the model outside of the script, for example in a python console? The following lines should work:

```py
from transformers import CTRLModel

model = CTRLModel.from_pretrained("ctrl", force_download=True)
```

The `force_download` option will re-download the file, this will confirm that the file being corrupted is not an issue.



Here the results:
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.platform()
'Linux-4.15.0-66-generic-x86_64-with-debian-buster-sid'
>>> platform.python_version()
'3.6.9'
>>> import transformers
>>> transformers.__version__
'2.1.1'
>>> import torch
>>> torch.__version__
'1.2.0'
>>> from transformers import CTRLModel
>>> model = CTRLModel.from_pretrained('ctrl', force_download=True)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 611/611 [00:00<00:00, 259358.34B/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6552025106/6552025106 [09:45<00:00, 11197418.01B/s]
>>> CTRLModel
<class 'transformers.modeling_ctrl.CTRLModel'>
>>> 
```

> Hi, you seem to be having problems when loading the CTRLModel in memory. Usually problems like this can occur if behind a firewall or if the file was corrupted. Could you try to load the model outside of the script, for example in a python console? The following lines should work:
> 
> ```python
> from transformers import CTRLModel
> 
> model = CTRLModel.from_pretrained("ctrl", force_download=True)
> ```
> 
> The `force_download` option will re-download the file, this will confirm that the file being corrupted is not an issue.