Fail in some tests (with detailed description)
Author: ghost
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
I have followed the README to install transformers.
However, there are some failures in the test. (mainly about multigpu_data_paralle_forward)
```
======================================================================================================================== short test summary info =========================================================================================================================
FAILED tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward - RuntimeError: tensor.ndimension() == static_cast<int64_t>(expected_size.size()) INTERNAL ASSERT FAILED at /pytorch/torch/csrc/cuda/comm.cpp:225, please report a bug to ...
FAILED tests/test_modeling_gpt2.py::GPT2ModelTest::test_multigpu_data_parallel_forward - RuntimeError: Gather got an input of invalid size: got [2, 2, 4, 7, 8], but expected [2, 4, 4, 7, 8] (gather at /pytorch/torch/csrc/cuda/comm.cpp:231)
FAILED tests/test_modeling_bart.py::BartHeadTests::test_lm_forward - timeout_decorator.timeout_decorator.TimeoutError: 'Timed Out'
FAILED tests/test_modeling_ctrl.py::CTRLModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA error: out of memory
FAILED tests/test_modeling_xlnet.py::XLNetModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA error: out of memory
FAILED tests/test_modeling_openai.py::OpenAIGPTModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 5.33 MiB already allocated; 17.19 MiB free; 6.00 MiB reserved in total ...
FAILED tests/test_modeling_mobilebert.py::MobileBertModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA error: out of memory
================================================================================================= 7 failed, 1066 passed, 562 skipped, 1396 warnings in 333.27s (0:05:33) =================================================================================================
```

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->


## Details
<!-- Description of your issue -->
I have tried many environments:
- conda: python3.7 + torch1.5.1 + cuda10.2 
- venv: python3.7 + torch1.5.1 + cuda10.2
- conda: python3.6 + torch1.4.0 + cuda10.1
- venv: python3.6 + torch1.4.0 + cuda10.1
- conda: python3.7 + torch1.5.1 + cuda10.2 + apex
- venv: python3.6 + torch1.4.0 + cuda10.1 + apex 

And I installed `transformers` **from source**.    
    
    
All of them failed in 5-7 tests.   
The `short test summary info` outputs of them are not all the same.    
But **all** of them are related to `test_multigpu_data_parallel_forward`.


One of the outputs of `transformers-cli env` is as the following     
(with the setting: venv: python3.6 + torch1.4.0 + cuda10.1 + apex):    
```
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 3.0.1
- Platform: Linux-4.15.0-1067-azure-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.11
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```
For my case (just fill out the two last points), 
```
- `transformers` version: 3.0.1
- Platform: Linux-4.15.0-1067-azure-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.11
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes
```
For this case, the `short test summary info` shows:   
```
======================================================================================================================== short test summary info =========================================================================================================================
FAILED tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward - RuntimeError: tensor.ndimension() == static_cast<int64_t>(expected_size.size()) INTERNAL ASSERT FAILED at /pytorch/torch/csrc/cuda/comm.cpp:225, please report a bug to ...
FAILED tests/test_modeling_gpt2.py::GPT2ModelTest::test_multigpu_data_parallel_forward - RuntimeError: Gather got an input of invalid size: got [2, 2, 4, 7, 8], but expected [2, 4, 4, 7, 8] (gather at /pytorch/torch/csrc/cuda/comm.cpp:231)
FAILED tests/test_modeling_bart.py::BartHeadTests::test_lm_forward - timeout_decorator.timeout_decorator.TimeoutError: 'Timed Out'
FAILED tests/test_modeling_ctrl.py::CTRLModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA error: out of memory
FAILED tests/test_modeling_xlnet.py::XLNetModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA error: out of memory
FAILED tests/test_modeling_openai.py::OpenAIGPTModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 5.33 MiB already allocated; 17.19 MiB free; 6.00 MiB reserved in total ...
FAILED tests/test_modeling_mobilebert.py::MobileBertModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA error: out of memory
================================================================================================= 7 failed, 1066 passed, 562 skipped, 1396 warnings in 333.27s (0:05:33) =================================================================================================
```
Some failures are about CUDA memory. 
Seems weird, especially the following one.
```
FAILED tests/test_modeling_openai.py::OpenAIGPTModelTest::test_multigpu_data_parallel_forward - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 5.33 MiB already allocated; 17.19 MiB free; 6.00 MiB reserved in total ...
```
So I checked the output of `nvidia-smi` **before and after** the test to make sure that **no other** processes hold memory.
```
Sat Jul  4 08:49:02 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000217:00:00.0 Off |                    0 |
| N/A   30C    P0    37W / 250W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | 00001C95:00:00.0 Off |                    0 |
| N/A   30C    P0    36W / 250W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-PCIE...  Off  | 0000735E:00:00.0 Off |                    0 |
| N/A   31C    P0    39W / 250W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-PCIE...  Off  | 0000AC50:00:00.0 Off |                    0 |
| N/A   31C    P0    36W / 250W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

```

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

The **details of the failures** are too long to post here.
Please refer to [this page](https://github.com/xx-zhou16/tmp_issue/blob/master/README.md) for details.   

## Related
I find a similar issue. #5070 
I think the only difference between his environment and mine is the version of transformers.
The failures of his and mine are both about `test_multigpu_data_parallel_forward`.
But #5070 has not been solved.

---- 
If you need me to provide other details, please reply to this issue. I will reply ASAP.   
I am focusing on this issue recently.    
I would be very glad if anyone can help solve this problem.



Comments:
It seems most of the tests that fail are due to `CUDA error: out of memory`?

> It seems most of the tests that fail are due to `CUDA error: out of memory`?

Yes. As the message shows, they are due to `CUDA error: out of memory`.    
But I am sure that **no** other processes are using CUDA **before and after** I run the test.   
My machine is with 4 Tesla V100 GPUs. And it works fine with other programs using both single and multi GPUs (e.g. fairseq).   
So I am confused with the error messages.    

Besides, there are some other error messages that show `invalid size` and `INTERNAL ASSERT FAILED`.

I have tried many environments. All of them fail in `test_multigpu_data_parallel_forward` due to **different** errors which are mainly `out of memory` and `invalid size`.    



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.