Unable to load pre-trained model/tokenizer when using Kubernetes
Author: g-karthik
Pull request: no

# 🐛 Bug

## Information

Model I am using: GPT-2

The problem arises when using:
* [Y] my own modified scripts: (give details below)

The tasks I am working on is:
* [Y] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a Kubernetes cluster, a Docker image with all needed dependencies, and a shared mount file-system for the cluster (the file-system contains the code and training data).
2. Create an `MPIJob` YAML file for running training.
3. Deploy the job on the cluster using `kubectl` or `helm`.

Here is the error I get (I have removed references to my custom code since that is irrelevant for this error):

```
Mon Jun 29 06:31:29 2020<stderr>:Traceback (most recent call last):
Mon Jun 29 06:31:29 2020<stderr>:  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 379, in _from_pretrained
Mon Jun 29 06:31:29 2020<stderr>:    resolved_vocab_files[file_id] = cached_path(file_path, cache_dir=cache_dir, force_download=force_download, proxies=proxies, resume_download=resume_download)
Mon Jun 29 06:31:29 2020<stderr>:  File "/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py", line 211, in cached_path
Mon Jun 29 06:31:29 2020<stderr>:    resume_download=resume_download, user_agent=user_agent)
Mon Jun 29 06:31:29 2020<stderr>:  File "/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py", line 312, in get_from_cache
Mon Jun 29 06:31:29 2020<stderr>:    os.makedirs(cache_dir)
Mon Jun 29 06:31:29 2020<stderr>:  File "/usr/lib/python3.6/os.py", line 220, in makedirs
Mon Jun 29 06:31:29 2020<stderr>:    mkdir(name, mode)
Mon Jun 29 06:31:29 2020<stderr>:FileExistsError: [Errno 17] File exists: '/root/.cache/torch/transformers'
Mon Jun 29 06:31:29 2020<stderr>:
Mon Jun 29 06:31:29 2020<stderr>:During handling of the above exception, another exception occurred:
Mon Jun 29 06:31:29 2020<stderr>:
Mon Jun 29 06:31:29 2020<stderr>:Traceback (most recent call last):
Mon Jun 29 06:31:29 2020<stderr>:    tokenizer = tokenizer_class.from_pretrained(args.model_checkpoint)
Mon Jun 29 06:31:29 2020<stderr>:  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 302, in from_pretrained
Mon Jun 29 06:31:29 2020<stderr>:    return cls._from_pretrained(*inputs, **kwargs)
Mon Jun 29 06:31:29 2020<stderr>:  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 391, in _from_pretrained
Mon Jun 29 06:31:29 2020<stderr>:    raise EnvironmentError(msg)
Mon Jun 29 06:31:29 2020<stderr>:OSError: Couldn't reach server at '{}' to download vocabulary files.
```

## Expected behavior

I wouldn't expect `FileExistsError: [Errno 17] File exists: '/root/.cache/torch/transformers'` to show up.

I also wouldn't expect `OSError: Couldn't reach server at '{}' to download vocabulary files.` to show up.

To debug whether there is a network issue with my containers causing the above `OSError`, I created a separate sample Pod and within the container, I started up python and tried to load a tokenizer, which worked just fine. So I don't think this is a network issue. I also don't quite understand why the server path is empty in the above log.

## Environment info
     
- `transformers` version: 2.3.0
- Platform: AWS Elastic Kubernetes Service (EKS) cluster
- Python version: 3.6
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes


Comments:
Do you have a code sample to reproduce this?

@LysandreJik sure, create a file named `train.py` containing only the following 2 lines:
```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
```

Create `helm/mpijob_chart/templates/job.yaml` as follows:

```
apiVersion: kubeflow.org/v1alpha2
kind: MPIJob
metadata:
  name: {{ .Values.jobName }}
spec:
  slotsPerWorker: {{ .Values.gpusPerNode }}
  cleanPodPolicy: Running
  backoffLimit: 0
  mpiReplicaSpecs:
    Worker:
      replicas: {{ .Values.nodeCount }}
      restartPolicy: Never
      template:
        spec:
          nodeSelector:
            beta.kubernetes.io/instance-type: {{ .Values.nodeType }}
          volumes:
          - name: fsx
            persistentVolumeClaim:
              claimName: fsx-claim
          containers:
          - image: {{ .Values.image }}
            imagePullPolicy: Always
            name: worker
            # This will allocate 8 GPUs for each worker, helping Kubernetes place resources.
            # Should match slotsPerWorker
            resources:
              limits:
                nvidia.com/gpu: {{ .Values.gpusPerNode }}
            # This mounts onto an external FSx volume, useful for storing the training data here.
            volumeMounts:
            - name: fsx
              mountPath: /fsx
            # This exposes the Docker container to listen to the mpirun ports
            env:
            - name: NCCL_SOCKET_IFNAME
              value: ^lo,docker0
            # Place the subdirectory on the Python path
            - name: PYTHONPATH
              value: {{ .Values.pythonPath }}
            # Some initial preparation, sleep for 14 days so the container will stay alive and respond to mpirun
            command: ["/bin/bash"]
            args: ["-c", "sleep 1209600"]
    Launcher:
      replicas: 1
      restartPolicy: Never
      template:
        spec:
          nodeSelector:
            beta.kubernetes.io/instance-type: {{ .Values.nodeType }}
          volumes:
          - name: fsx
            persistentVolumeClaim:
              claimName: fsx-claim
          containers:
          - image: {{ .Values.image }}
            imagePullPolicy: Always
            name: launcher
            volumeMounts:
            - mountPath: /fsx
              name: fsx
            # Wait 15 seconds so any additional dependencies being manually installed will finish on worker nodes
            command: ["/bin/bash"]
            args:
            - "-c"
            - "sleep 15 && \
                mpirun -np {{ mul .Values.nodeCount .Values.gpusPerNode }} \
                --allow-run-as-root \
                --timestamp-output \
                -bind-to none -map-by slot \
                -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
                -mca pml ob1 -mca btl ^openib \
                {{ .Values.command1 }} "
```

`helm/mpijob_chart/values.yaml` for the above `job.yaml`:

```
# Each job name must be unique to the Kubernetes cluster
jobName: gpt2-training
nodeCount: 1
# If gpusPerNode is too high, the job will pend forever.
gpusPerNode: 7
nodeType: p3dn.24xlarge
image: <REPLACE WITH YOUR OWN IMAGE HERE, I suggest building a custom image from Horovod base image with transformers installed in it>
pythonPath: /fsx/myCodeRepo
command1: "python /fsx/myCodeRepo/train.py &> /fsx/testout.txt"
```

For setting up the shared file-system for the cluster (i.e., FSx, the persistent volume and persistent volume claim YAMLs), the instructions/templates are here: https://github.com/kubernetes-sigs/aws-fsx-csi-driver/tree/master/examples/kubernetes/static_provisioning

Copy `train.py` to `/fsx/myCodeRepo/` in the shared file-system. Then you can use `helm install trainjob helm/mpijob_chart` to run the job.

Have you tried with changing the `cache_dir` when using `from_pretrained`?

@LysandreJik yes, I changed it to:

```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2", cache_dir="/fsx/transformers_cache/")
```

The error is still the same. Instead of giving `FileExistsError ` with `/root/.cache/torch/transformers`, it gives the error with `/fsx/transformers_cache/`. And the `OSError` is also thrown just like before.

@LysandreJik it seems that this error only happens when I have multiple processes during training. I tried a simple test of using 1 node and only 1 GPU in that node during training, and that is working totally fine.

UPDATE: So as a temporary workaround, what I did was I kicked off a job with 1 node, 1 GPU, allowing for the pre-trained tokenizer/model caching to happen fine. Then I killed the job immediately, and kicked off a new one with N nodes, 8 GPUs. No errors this time. But this is kinda a hack. Do you have a better solution?