Can't Select Specific GPU by TrainingArguments
Author: EasonC13
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: Jupyter Notebook on Ubuntu
- Python version: 3.7
- PyTorch version (GPU?): 1.8.0+cu111
- Using GPU in script?: No, By Jupyter Notebook
- Using distributed or parallel set-up in script?:It is distributed but I don't want that

### Who can help
- trainer: @sgugger
find by git-blame: @philschmid


## To reproduce

By TrainingArguments, I want to set up my compute device only to torch.device(type='cuda', index=1).

If I not set local_rank when init TrainingArguments, it will compute on both GPU.

Steps to reproduce the behavior:

```
from transformers import TrainingArguments, Trainer, EvalPrediction

training_args = TrainingArguments(
    learning_rate=1e-4,
    num_train_epochs=6,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    logging_steps=200,
    output_dir="./training_output",
    overwrite_output_dir=True,
    # The next line is important to ensure the dataset labels are properly passed to the model
    remove_unused_columns=False,
    local_rank= 1
)
```

Then you will get `ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set`

But after I set 

```
import os
os.environ["RANK"]="1"
```

I get `ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable WORLD_SIZE expected, but not set`

These error not happen if I not set local_rank when init TrainingArguments even though I don't set any environment variable.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I want to set up my compute device only to torch.device(type='cuda', index=1).

Comments:
You should use the env variable `CUDA_VISIBLE_DEVICES` to set the GPUs you want to use. If you have multiple GPUs available, the `Trainer` will use all of them, that is expected and not a bug.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I have a similar challenge. I have 3 GPUs in the server. I run the script like this:
```
CUDA_VISIBLE_DEVICES=2 python main.py.
```
however, when I print `training_args.device`, it still shows cuda:0. `model.device` shows the same thing
this does not help either:
```
export CUDA_VISIBLE_DEVICES=2
python main.py
```

I am using Seq2SeqTrainingArguments

This is normal, PyTorch names all visible devices from 0 to the number -1. So cuda0 in PyTorch is the first device you set as available, in this case GPU 2.

I'm having the same issue with the Trainer class.

Even after setting CUDA_VISIBLE_DEVICES, it still attempts to use all GPUs on my machine. This is problematic, as I share this server with other users. (And even during times with open GPUs, there are more than 8 GPUS present and it exhausts peer mapping resources.)

Error can be reproduced using the LED fine-tune Colab notebook demo if downloaded to a multi-GPU machine. https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb

with the following added code inserted in the first cell:

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"

I've attempted setting the CUDA_VISIBLE_DEVICES environmental variable other ways (ie. from terminal, etc.) with similar results.

I've also attempted using the PyTorch method of specifying GPU, with similar results:
import torch
DEFAULT_DEVICE = "cuda"
torch.cuda.set_device(0)

No you need to set that environment variable with the launch command, not inside your training script:
```
CUDA_VISIBLE_DEVICES="0" python main.py
```

So is there any way to do this within a notebook?

You need to set the variable before launching the jupyter notebook
```
CUDA_VISIBLE_DEVICES="0" jupyter notebook
```

Ahhh, thank you. That successfully restricts the GPUs accessed in the notebook. 

> You need to set the variable before launching the jupyter notebook
> 
> ```
> CUDA_VISIBLE_DEVICES="0" jupyter notebook
> ```

It's very inconvenient each time to restart jupyter lab/notebook to just change the device. Also, I may want to use several notebooks on different devices. PytorchLightening, for example, gives you freedom to select device for each run.

In Jupyter Notebook, we can use one of these:

```
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0"
```

```
%env CUDA_DEVICE_ORDER=PCI_BUS_ID
%env CUDA_VISIBLE_DEVICES=0
```

Referring to all above solutions, all my GPUs are running or get CUDA device errors.
As alternatives, I override TrainingArguments Class. However, it might have undiscovered issues though.
* backgrounds : I have more than one GPUs. Using huggingface trainer, all devices are involved in training.
* problems : Trainer [seems to use ddp after checking device and n_gpus](https://github.com/huggingface/transformers/blob/a22db885b41b3a1b302fc206312ee4d99cdf4b7c/src/transformers/trainer.py#L1159) method in TrainingArugments , and `_setup_devices` in [TrainingArguments](https://github.com/huggingface/transformers/blob/a22db885b41b3a1b302fc206312ee4d99cdf4b7c/src/transformers/training_args.py#L1105)  controls overall device setting. 
* temporary remedies : Instead of overriding `_setup_devices` (since it relates multiple dependent functions), I manually set `device` method and `n_gpus` method. In this case, I don't need to give any `os.environ` or `CUDA_VISIBLE_DEVICES `in front of python commands for single use. However, it may require if you want to use selected two or three gpus out of 4.

```
class customTrainingArguments(TrainingArguments):
    def __init__(self,*args, **kwargs):
        super(customTrainingArguments, self).__init__(*args, **kwargs)

    @property
    @torch_required
    def device(self) -> "torch.device":
        """
        The device used by this process.
        Name the device the number you use.
        """
        return torch.device("cuda:3")

    @property
    @torch_required
    def n_gpu(self):
        """
        The number of GPUs used by this process.
        Note:
            This will only be greater than one when you have multiple GPUs available but are not using distributed
            training. For distributed training, it will always be 1.
        """
        # Make sure `self._n_gpu` is properly setup.
        # _ = self._setup_devices
        # I set to one manullay
        self._n_gpu = 1
        return self._n_gpu
```

In a related problem, CUDA_VISIBLE_DEVICES doesn't seem to work, as I set it to use only the second gpu, but it always uses the first. I tried @kimcando 's solution but it just gives another error: "module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1", even after sending the data to device cuda:1. 

@hrmello have you been able to solve this problem? I'm facing the same issue

> @hrmello have you been able to solve this problem? I'm facing the same issue

@ngonhi Unfortunately not. I found out that if you don't specify the GPU, it finds and use all of them. But if you only want to run your code using a single one, you must use gpu 0. 

It feels folks need this feature so might be worth reopening the issue. @sgugger?

We can reopen it, but there is nothing I can do to fix it as it is part of the launching process of your script, which is implemented in PyTorch, not in Transformers :man_shrugging: 

We are implementing this option in the `accelerate launcher` [here](https://github.com/huggingface/accelerate/pull/732) for folks interested.

Thank you. This is very helpful. 

> Referring to all above solutions, all my GPUs are running or get CUDA device errors. As alternatives, I override TrainingArguments Class. However, it might have undiscovered issues though.
> 
> * backgrounds : I have more than one GPUs. Using huggingface trainer, all devices are involved in training.
> * problems : Trainer [seems to use ddp after checking device and n_gpus](https://github.com/huggingface/transformers/blob/a22db885b41b3a1b302fc206312ee4d99cdf4b7c/src/transformers/trainer.py#L1159) method in TrainingArugments , and `_setup_devices` in [TrainingArguments](https://github.com/huggingface/transformers/blob/a22db885b41b3a1b302fc206312ee4d99cdf4b7c/src/transformers/training_args.py#L1105)  controls overall device setting.
> * temporary remedies : Instead of overriding `_setup_devices` (since it relates multiple dependent functions), I manually set `device` method and `n_gpus` method. In this case, I don't need to give any `os.environ` or `CUDA_VISIBLE_DEVICES `in front of python commands for single use. However, it may require if you want to use selected two or three gpus out of 4.
> 
> ```
> class customTrainingArguments(TrainingArguments):
>     def __init__(self,*args, **kwargs):
>         super(customTrainingArguments, self).__init__(*args, **kwargs)
> 
>     @property
>     @torch_required
>     def device(self) -> "torch.device":
>         """
>         The device used by this process.
>         Name the device the number you use.
>         """
>         return torch.device("cuda:3")
> 
>     @property
>     @torch_required
>     def n_gpu(self):
>         """
>         The number of GPUs used by this process.
>         Note:
>             This will only be greater than one when you have multiple GPUs available but are not using distributed
>             training. For distributed training, it will always be 1.
>         """
>         # Make sure `self._n_gpu` is properly setup.
>         # _ = self._setup_devices
>         # I set to one manullay
>         self._n_gpu = 1
>         return self._n_gpu
> ```

It works. I just have to comment out the `@torch_required` and add `import torch` at line 1, then I can freely choose whatever GPU I want. Thanks a million.

Thank you @kimcando I finally got my code to run on a single, specified GPU with your modification! 

> Referring to all above solutions, all my GPUs are running or get CUDA device errors. As alternatives, I override TrainingArguments Class. However, it might have undiscovered issues though.
> 
> * backgrounds : I have more than one GPUs. Using huggingface trainer, all devices are involved in training.
> * problems : Trainer [seems to use ddp after checking device and n_gpus](https://github.com/huggingface/transformers/blob/a22db885b41b3a1b302fc206312ee4d99cdf4b7c/src/transformers/trainer.py#L1159) method in TrainingArugments , and `_setup_devices` in [TrainingArguments](https://github.com/huggingface/transformers/blob/a22db885b41b3a1b302fc206312ee4d99cdf4b7c/src/transformers/training_args.py#L1105)  controls overall device setting.
> * temporary remedies : Instead of overriding `_setup_devices` (since it relates multiple dependent functions), I manually set `device` method and `n_gpus` method. In this case, I don't need to give any `os.environ` or `CUDA_VISIBLE_DEVICES `in front of python commands for single use. However, it may require if you want to use selected two or three gpus out of 4.
> 
> ```
> class customTrainingArguments(TrainingArguments):
>     def __init__(self,*args, **kwargs):
>         super(customTrainingArguments, self).__init__(*args, **kwargs)
> 
>     @property
>     @torch_required
>     def device(self) -> "torch.device":
>         """
>         The device used by this process.
>         Name the device the number you use.
>         """
>         return torch.device("cuda:3")
> 
>     @property
>     @torch_required
>     def n_gpu(self):
>         """
>         The number of GPUs used by this process.
>         Note:
>             This will only be greater than one when you have multiple GPUs available but are not using distributed
>             training. For distributed training, it will always be 1.
>         """
>         # Make sure `self._n_gpu` is properly setup.
>         # _ = self._setup_devices
>         # I set to one manullay
>         self._n_gpu = 1
>         return self._n_gpu
> ```

This is the **best** solution for now, I would like to provide more usage for new bees,
(we should comment out `@torch_required`.
We can specify the GPU by changing the `return torch.device("cuda:3")` in the `def device(self)`,

After overloading class `customTrainingArguments`, 
we only need to `training_args = customTrainingArguments(...)` **instead** of `training_args = TrainingArguments(...)`
the arguments inside are as usual 
It is the **simplest** way now 

> This is the best solution for now, I would like to provide more usage for new bees,
(we should comment out @torch_required.
We can specify the GPU by changing the return torch.device("cuda:3") in the def device(self),

> After overloading class customTrainingArguments,
we only need to training_args = customTrainingArguments(...) instead of training_args = TrainingArguments(...)
the arguments inside are as usual
It is the simplest way now

Which version of transformers is it working with?

I get error 
```
AttributeError: 'CustomTrainingArguments' object has no attribute 'distributed_state'
```


I follow this tutorial to finetune Whisper, however, the code won't select specific GPUs and only run on gpu:0.
I used CUDA_VISIBLE_DEVICES.

code used
`from transformers import Seq2SeqTrainingArguments
from transformers import Seq2SeqTrainer`

best regards

> > This is the best solution for now, I would like to provide more usage for new bees,
> > (we should comment out @torch_required.
> > We can specify the GPU by changing the return torch.device("cuda:3") in the def device(self),
> 
> > After overloading class customTrainingArguments,
> > we only need to training_args = customTrainingArguments(...) instead of training_args = TrainingArguments(...)
> > the arguments inside are as usual
> > It is the simplest way now
> 
> Which version of transformers is it working with?
> 
> I get error
> 
> ```
> AttributeError: 'CustomTrainingArguments' object has no attribute 'distributed_state'
> ```

Did you ever fixed this error?

EDIT: Seems like using an older version worked here, Version 4.29.2 worked 

from accelerate.state import AcceleratorState
from accelerate.utils import DistributedType

class cached_property(property):
    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError("unreadable attribute")
        attr = "__cached_" + self.fget.__name__
        cached = getattr(obj, attr, None)
        if cached is None:
            cached = self.fget(obj)
            setattr(obj, attr, cached)
        return cached

class CustomTrainingArguments(TrainingArguments):
    def __init__(self,*args, **kwargs):
        super(CustomTrainingArguments, self).__init__(*args, **kwargs)

    @property
    def device(self) -> "torch.device":
        return torch.device("cuda:0")

    @property
    def n_gpu(self):
        self._n_gpu = 1
        return self._n_gpu

    @property
    def parallel_mode(self):
        return "not_parallel"
    
    @cached_property
    def _setup_devices(self) -> "torch.device":
        self.distributed_state = AcceleratorState(backend=self.ddp_backend)
        self._n_gpu = 1
        device = self.distributed_state.device
        self.local_rank = self.distributed_state.local_process_index
        self.distributed_state.distributed_type = DistributedType.NO
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        torch.cuda.set_device(device)
        return device`

Any update on this? I'm trying to use batch size rampup, where the first batch size of 1, 2nd epoch a batch size of 2, etc. Currently it seems the official way is to run the script 3 times if batch size is under number of gpus