About Multi GPU
Author: HyeyeonKoo
Pull request: no

## Environment info
- `transformers` version: 3.5.0
- Platform: Linux-3.10.0-514.el7.x86_64-x86_64-with-centos-7.3.1611-Core
- Python version: 3.6.4
- PyTorch version (GPU?): 1.7.0
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: Y

### Who can help
@LysandreJik,  @sgugger
 
## Information
Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. code

```
from transformers import RobertaConfig
config = RobertaConfig(
    vocab_size=34492,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
    type_vocab_size=1,
    position_embedding_type="absolute"
)

from transformers import RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained("tokenizer", max_len=512)

from transformers import RobertaForMaskedLM
model = RobertaForMaskedLM(config=config)

from datetime import datetime
from transformers import LineByLineTextDataset
train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="train.txt",
    block_size=tokenizer.max_len_single_sentence
)
eval_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="eval.txt",
    block_size=tokenizer.max_len_single_sentence
)

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

from transformers import Trainer, TrainingArguments
training_args = TrainingArguments(
    output_dir="output",
    overwrite_output_dir=True,
    do_train=True,
    do_eval=True,
    evaluation_strategy="epoch",
    learning_rate=6e-4,
    adam_beta1=0.9,
    adam_beta2=0.98,
    adam_epsilon=1e-6,
    per_device_train_batch_size=200,
    per_device_eval_batch_size=200,
    num_train_epochs=14,
    disable_tqdm=True
)
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()
```

2. log
```
True
0: 1
0: Tesla V100-PCIE-32GB
1: True
1: 1
1: Tesla V100-PCIE-32GB
0: build dataset : 0:00:07.309346
0: build dataset : 0:00:00.328179
1: build dataset : 0:00:07.945437
1: build dataset : 0:00:00.364413
0: {'loss': 10.5927734375, 'learning_rate': 0.0005999489187808615, 'epoch': 0.0011918951132300357}
1: {'loss': 10.57164478302002, 'learning_rate': 0.0005999489187808615, 'epoch': 0.0011918951132300357}
0: {'loss': 6.9972802049411325, 'learning_rate': 0.000574459390430785, 'epoch': 0.5959475566150179}
1: {'loss': 7.002058581503216, 'learning_rate': 0.000574459390430785, 'epoch': 0.5959475566150179}
0: {'eval_loss': 7.694924354553223, 'epoch': 1.0}
1: {'eval_loss': 7.6843767166137695, 'epoch': 1.0}
0: {'loss': 6.86546826171875, 'learning_rate': 0.0005489187808615699, 'epoch': 1.1918951132300357}
1: {'loss': 6.86378662109375, 'learning_rate': 0.0005489187808615699, 'epoch': 1.1918951132300357}
0: {'loss': 6.84359375, 'learning_rate': 0.0005233781712923548, 'epoch': 1.7878426698450536}
1: {'loss': 6.842060546875, 'learning_rate': 0.0005233781712923548, 'epoch': 1.7878426698450536}
0: {'eval_loss': 7.635812759399414, 'epoch': 2.0}
1: {'eval_loss': 7.633483409881592, 'epoch': 2.0}
0: {'loss': 6.812291015625, 'learning_rate': 0.0004978375617231397, 'epoch': 2.3837902264600714}
1: {'loss': 6.811927734375, 'learning_rate': 0.0004978375617231397, 'epoch': 2.3837902264600714}
0: {'loss': 6.8180390625, 'learning_rate': 0.0004722969521539247, 'epoch': 2.9797377830750893}
1: {'loss': 6.8180390625, 'learning_rate': 0.0004722969521539247, 'epoch': 2.9797377830750893}
0: {'eval_loss': 7.621339797973633, 'epoch': 3.0}
1: {'eval_loss': 7.620441436767578, 'epoch': 3.0}
0: {'loss': 6.8016015625, 'learning_rate': 0.0004467563425847096, 'epoch': 3.575685339690107}
1: {'loss': 6.8015546875, 'learning_rate': 0.0004467563425847096, 'epoch': 3.575685339690107}
0: {'eval_loss': 7.575932025909424, 'epoch': 4.0}
1: {'eval_loss': 7.5758209228515625, 'epoch': 4.0}
0: {'loss': 6.81323828125, 'learning_rate': 0.0004212157330154946, 'epoch': 4.171632896305125}
1: {'loss': 6.81312109375, 'learning_rate': 0.0004212157330154946, 'epoch': 4.171632896305125}
0: {'loss': 6.80004296875, 'learning_rate': 0.0003956751234462795, 'epoch': 4.767580452920143}
1: {'loss': 6.80001953125, 'learning_rate': 0.0003956751234462795, 'epoch': 4.767580452920143}
0: {'eval_loss': 7.579530715942383, 'epoch': 5.0}
1: {'eval_loss': 7.579504013061523, 'epoch': 5.0}
0: {'loss': 6.79704296875, 'learning_rate': 0.0003701345138770645, 'epoch': 5.363528009535161}
1: {'loss': 6.79696875, 'learning_rate': 0.0003701345138770645, 'epoch': 5.363528009535161}
0: {'loss': 6.796515625, 'learning_rate': 0.0003445939043078495, 'epoch': 5.959475566150179}
1: {'loss': 6.79640234375, 'learning_rate': 0.0003445939043078495, 'epoch': 5.959475566150179}
0: {'eval_loss': 7.59311580657959, 'epoch': 6.0}
1: {'eval_loss': 7.593157768249512, 'epoch': 6.0}
0: {'loss': 6.7975078125, 'learning_rate': 0.0003190532947386344, 'epoch': 6.5554231227651965}
1: {'loss': 6.7974375, 'learning_rate': 0.0003190532947386344, 'epoch': 6.5554231227651965}
0: {'eval_loss': 7.5591912269592285, 'epoch': 7.0}
1: {'eval_loss': 7.559223175048828, 'epoch': 7.0}
0: {'loss': 6.8036171875, 'learning_rate': 0.00029351268516941936, 'epoch': 7.151370679380214}
1: {'loss': 6.803546875, 'learning_rate': 0.00029351268516941936, 'epoch': 7.151370679380214}
0: {'loss': 6.79696875, 'learning_rate': 0.0002679720756002043, 'epoch': 7.747318235995232}
1: {'loss': 6.7969921875, 'learning_rate': 0.0002679720756002043, 'epoch': 7.747318235995232}
0: {'eval_loss': 7.575222492218018, 'epoch': 8.0}
1: {'eval_loss': 7.574929714202881, 'epoch': 8.0}
0: {'loss': 6.796890625, 'learning_rate': 0.00024243146603098925, 'epoch': 8.34326579261025}
1: {'loss': 6.7968515625, 'learning_rate': 0.00024243146603098925, 'epoch': 8.34326579261025}
0: {'loss': 6.788359375, 'learning_rate': 0.00021689085646177421, 'epoch': 8.939213349225268}
1: {'loss': 6.788375, 'learning_rate': 0.00021689085646177421, 'epoch': 8.939213349225268}
0: {'eval_loss': 7.567000389099121, 'epoch': 9.0}
1: {'eval_loss': 7.566658973693848, 'epoch': 9.0}
0: {'loss': 6.794640625, 'learning_rate': 0.00019135024689255915, 'epoch': 9.535160905840286}
1: {'loss': 6.7945859375, 'learning_rate': 0.00019135024689255915, 'epoch': 9.535160905840286}
0: {'eval_loss': 7.5506415367126465, 'epoch': 10.0}
1: {'eval_loss': 7.550570487976074, 'epoch': 10.0}
0: {'loss': 6.78496875, 'learning_rate': 0.0001658096373233441, 'epoch': 10.131108462455304}
1: {'loss': 6.7848984375, 'learning_rate': 0.0001658096373233441, 'epoch': 10.131108462455304}
0: {'loss': 6.7898984375, 'learning_rate': 0.00014026902775412904, 'epoch': 10.727056019070321}
1: {'loss': 6.7898203125, 'learning_rate': 0.00014026902775412904, 'epoch': 10.727056019070321}
0: {'eval_loss': 7.568336486816406, 'epoch': 11.0}
1: {'eval_loss': 7.568056583404541, 'epoch': 11.0}
0: {'loss': 6.79440625, 'learning_rate': 0.000114728418184914, 'epoch': 11.32300357568534}
1: {'loss': 6.7943984375, 'learning_rate': 0.000114728418184914, 'epoch': 11.32300357568534}
0: {'loss': 6.78665625, 'learning_rate': 8.918780861569896e-05, 'epoch': 11.918951132300357}
1: {'loss': 6.786703125, 'learning_rate': 8.918780861569896e-05, 'epoch': 11.918951132300357}
0: {'eval_loss': 7.579376220703125, 'epoch': 12.0}
1: {'eval_loss': 7.5791497230529785, 'epoch': 12.0}
0: {'loss': 6.79565625, 'learning_rate': 6.364719904648391e-05, 'epoch': 12.514898688915375}
1: {'loss': 6.795640625, 'learning_rate': 6.364719904648391e-05, 'epoch': 12.514898688915375}
0: {'eval_loss': 7.5773115158081055, 'epoch': 13.0}
1: {'eval_loss': 7.577144622802734, 'epoch': 13.0}
0: {'loss': 6.795859375, 'learning_rate': 3.810658947726885e-05, 'epoch': 13.110846245530393}
1: {'loss': 6.795796875, 'learning_rate': 3.810658947726885e-05, 'epoch': 13.110846245530393}
0: {'loss': 6.79365625, 'learning_rate': 1.2565979908053806e-05, 'epoch': 13.70679380214541}
1: {'loss': 6.793703125, 'learning_rate': 1.2565979908053806e-05, 'epoch': 13.70679380214541}
0: {'eval_loss': 7.550729751586914, 'epoch': 14.0}
0: {'epoch': 14.0}
0: train time : 2:00:32.638885
1: {'eval_loss': 7.550601482391357, 'epoch': 14.0}
1: {'epoch': 14.0}
1: train time : 2:00:54.112366
```

## Expected behavior
Hello! I am not sure this is BUG, but I don't know where I can ask a question about this. So, if it is not appropriate, please tell me how can I get the answer about this.
I wrote the code like above and I have two GPUs. I understand that transformers automatically allocate data to each GPU, so I don't need to set up anything in code. But, the log seems like each GPU train separate model. I expect that each GPU is trained by scattered data, and gather the loss. However, GPU0's loss and GPU1's loss have same values. Also, there is no time different between using two gpus(7302sec) and single gpu(7252sec). Is there anything I can do for reducing training time with two GPUs?    

Comments:
Hi there, those kinds of questions should be asked on the [forums](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests only, so closing this.

For a quick answer (move the discussion to the forums for a longer discussion!), it's normal that each GPU takes a slightly different time to train as all CUDA operations are asynchronous and the program is launched twice in parallel to be executed by both. It's also normal to see two different losses as the loss is not gathered across devices during training, only the gradients.

Thank you for the answer. I will move this to forums.