[scripts] Proposal: add a specific device flag
Author: julien-c
Pull request: yes
Labels: name: wontfix

wdyt?

Will do in other scripts if this gets merged.

My use case is I have an instance with multiple GPUs and want to run one generation on `cuda:0`, another one on `cuda:1`, etc.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=h1) Report
> Merging [#1603](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e16d46843a19ab289b82138e4eccec5610a76de7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1603/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1603   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1603/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9jdHJsLnB5) | `96.03% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=footer). Last update [e16d468...b0af23c](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


In my script, I take another approach for this. I assume that distributed training can only be instantiated by using torch's launch script 'or that at least the WORLD_SIZE env variable is set). `local_rank` will be used for the GPU id, even when not in distributed mode.

```python
# torch.distributed.launch adds a world_size environment variable
distributed = int(os.environ['WORLD_SIZE']) > 1 if 'WORLD_SIZE' in os.environ else False
```

Based on that, you can decide what you want to do with `local_rank`. If we're in distributed mode, start the process group, if we're not: use the `local_rank` cuda device.

```python
if local_rank == -1 or not torch.cuda.is_available():
    device = torch.device('cpu')
else:
    device = torch.device(f"cuda:{local_rank}")
    if distributed:
        dist.init_process_group(backend='nccl', init_method='env://')
```

As a bonus, to ensure that all processes such as validating only happen on the main device, even if that's not cuda:0 (even though personally I do that on all devices, too):

```python
is_first_process = not distributed or local_rank in [0, -1]
# ...
if args.do_eval and is_first_process:
    # do eval
```

I merely post this for possible inspiration, of course!

Sounds good to me let's add this to all the examples (and the template in `templates/adding_a_new_example_script`)

> Sounds good to me let's add this to all the examples

The other scripts maybe make less sense as you would want to train on all available devices? Not 100% sure yet.

Ok I see, then maybe let's have the device flag on `run_generation` instead of `run_squad` (as currently proposed in the PR)?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.