Trainer.train() gets stuck when executed on K8 pods
Author: pthalasta
Pull request: no

## Environment info

- `transformers` version: 4.3.2
- Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

@LysandreJik @sgugger

## Information

Model I am using BertForSequenceClassification

The problem arises when using:
* [ ] my own modified scripts: (give details below)
When i try to start the training of the model in K8 pod (in kubeflow env) with ubuntu 18.04 image. There is no output or error shown by the function even after 30mins of runtime. GPU usage doesn't change as well

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   28C    P0    56W / 300W |   1514MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
| N/A   27C    P0    56W / 300W |   1134MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

```


## Expected behavior
Model to train and output the results

Comments:
Hi there. Unless you tell us what script you are using and how you are launching it, there is nothing we can do to help.

@sgugger i'm trying to execute this example script
https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb

This notebook works fine in a docker container but not on K8s pod

Script is launched in jupyter server that is hosted on kubeflow

@sgugger any information on how i can solve the issue?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.