run_generation.py Runtime error
Author: Yondijr
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

**Hi everyone,** 
I'm  currently trying to execute the run_generation example script.
 However I can not get it to work. Tried to reinstall and played around with the parameters. However the error I get is always the same:

**I'm running the example from the website:** 

_python run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=gpt2_

**It allocates resources etc. However when it is starting to generate I get this error:**

_Namespace(device=device(type='cuda'), length=20, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_samples=1, padding_text='', prompt='', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
  0%|                                                                                                                               | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_generation.py", line 264, in <module>
    main()
  File "run_generation.py", line 249, in main
    device=args.device,
  File "run_generation.py", line 142, in sample_sequence
    outputs = model(**inputs)  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
  File "/home/gstein/myPython/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/gstein/Executables/transformers/transformers/modeling_gpt2.py", line 533, in forward
    head_mask=head_mask)
  File "/home/gstein/myPython/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/gstein/Executables/transformers/transformers/modeling_gpt2.py", line 373, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0]_


## Environment

* OS: Ubuntu 16.04.3
* Python version: 3.5.2
* Torch version 1.0
* cloned from Git 4. November 2019:
* Using GPU: Yes/No 

##Thoughts?
Does anyone else encounter this behavior? 
Or even better: Does somebody has a solution for it?) 


Comments:
### Description
Running `python run_generation.py 
--model_type=gpt2 
--model_name_or_path=gpt2` in my environment works as expected.

My suggestions:
- update PyTorch to latest version with `pip install --upgrade torch`
- try to install Transformers' library from branch master, and not use the one installed with PyPi
- try to use the CPU for inference (typically, we'll use GPU/TPU only in training a model, whereas in inference mode we'll use CPU) --> add **--no_cuda** when calling `run_generation.py` script

### My environment
- __Python__ '3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) \n[GCC 7.3.0]'
- __PyTorch__ v1.3.0
- __HuggingFace's Transformers__ v2.1.1 (installed **not** from PyPi, but with **git clone from master branch**)
- __O.S.__ 'Linux-4.15.0-66-generic-x86_64-with-debian-buster-sid'
- __Device__ tried with CPU and GPU (both work as expected)

### Output Example
```2019-11-04 14:16:59.671926: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-11-04 14:16:59.684347: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-04 14:16:59.684998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce GTX 980 Ti major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:01:00.0
2019-11-04 14:16:59.685168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-11-04 14:16:59.686072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-11-04 14:16:59.686812: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-11-04 14:16:59.686987: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-11-04 14:16:59.687996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-11-04 14:16:59.688758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-11-04 14:16:59.688840: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
2019-11-04 14:16:59.688849: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-11-04 14:16:59.689057: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-04 14:16:59.712088: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3408000000 Hz
2019-11-04 14:16:59.712609: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x561d09196760 executing computations on platform Host. Devices:
2019-11-04 14:16:59.712622: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-11-04 14:16:59.752521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-04 14:16:59.753095: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x561d09161190 executing computations on platform CUDA. Devices:
2019-11-04 14:16:59.753111: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 980 Ti, Compute Capability 5.2
2019-11-04 14:16:59.753166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-04 14:16:59.753173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      
11/04/2019 14:17:00 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /home/vidiemme/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
11/04/2019 14:17:00 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /home/vidiemme/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
11/04/2019 14:17:01 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /home/vidiemme/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
11/04/2019 14:17:01 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "is_decoder": false,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

11/04/2019 14:17:01 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /home/vidiemme/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
11/04/2019 14:17:12 - INFO - __main__ -   Namespace(device=device(type='cuda'), length=20, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_samples=1, padding_text='', prompt='', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
Model prompt >>> Hi. I'm Edward and I'm a journalist.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 27.07it/s]
 This writer lost her health and is now ill. But I want to show you something. That's
Model prompt >>> 
```

@Yondijr could you try to update your Torch version to a more recent version than 1.0 and let us know if it fixes the issue?

@LysandreJik 
Yes!
However it **ONLY** works with  torch 1.3 with GPU support. Interestingly even with the 1.3-cpu version I got the same error.
Thanks for the tip :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.