finetuning T5 on translation on TPU, questions about clarifying the setup
Author: rabeehkarimimahabadi
Pull request: no

Hi,
I'd like to run finetune.py with wmt dataset in TPU to train from scratch not finetune. I appreciate the response to some questions:
1) Why there are two versions of fine-tuning using Seq2SeqTrainer and finetune.py and which one is suitable for my usecase?
2) Seq2SeqTrainer does not support predict on TPU, is this the case for finetune.py as well? 
3) when running on TPU, it is written to use  xla_spawn.py, since  the codes of finetune.py are with pytorch lightening, is this needed to launch them with xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)? 
4) in the finetune.py dataloader, I see this is distributed based on number of gpus, but I cannot see this is distributed also when one uses tpus, Is this taking care of making dataloader distributed in case of using TPU automatically?
5) If using finetune.py with TPU, is there any specific setup I need to add for finetuning/training T5 on WMT?
6) I assume one need to use maybe something like this as a sampler for TPU distributed dataloaders, I see this is not the case in the codes of finetune.py, does data parallelism works in finetune.py? 
7) when to use sortish_sampler/dynamic_sampler, do they work on TPUs?
```
  sampler = torch.utils.data.distributed.DistributedSampler(
        dataset,
        num_replicas=xm.xrt_world_size(),
        rank=xm.get_ordinal(),
        shuffle=True)

```
8. fp16 does it work with TPUs as well? 
9. with iterable datasets seems dataloader in finetune_trainer is not working with distributed detup on TPU, do you know how to implement it? am I missing something? thanks 
thank you very much.

Comments:
in the main page of examples you mention one can pass any script to run it on tpu, but inside the seq2seq it seems one needs to use finetune_trainer and not finetune.py for tpus, I am confused which one to use, thanks for your help

@sshleifer @patil-suraj Maybe we could improve the documentation here

Hi, thank you @LysandreJik, do you know which version of finetune.py to finetune_trainer.py are working with tpus? in the documentation it is written any example can be run with multiple tpus by using xla_spawn.py but I am not sure if this is true for finetune.py too. thanks 

`finetune_trainer.py` works with TPU, here is [the wmt script](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/builtin_trainer/finetune_tpu.sh)


1) They were developed at different times. We are trying to get them both working well.
2) finetune.py should not be used with TPU at all.
3) yes, see script.
4) see script
5)  see script
6) see script
7) I would guess that those samplers don't work on TPU.
8) No it does not.
9) No idea, maybe @patil-suraj knows.




thanks Sam for this, so finetune.py does not work? In the documentation,
this is written one can run all examples with xla_spawn on TPU, I
appreciate updating the README mentioning it.
thank you.

On Wed, Nov 4, 2020 at 8:06 PM Sam Shleifer <notifications@github.com>
wrote:

> finetune_trainer.py works with TPU, cc @patil-suraj
> <https://github.com/patil-suraj> .
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8290#issuecomment-721917741>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHH345RVTXO3EA7ZUPUTSOGQ3XANCNFSM4TKBDLQQ>
> .
>


Correct, updated docs.

thank you Sam

On Wed, Nov 4, 2020 at 9:46 PM Sam Shleifer <notifications@github.com>
wrote:

> Correct, updated docs.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8290#issuecomment-721966350>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHH4QZWN6KEL2BKSF7BTSOG4TVANCNFSM4TKBDLQQ>
> .
>