Exception: process 0 terminated with exit code 17 when using xla_spawn 
Author: rabeehkarimimahabadi
Pull request: no

Hi
I am calling finetune_trainer with xla_spawn command, on TPU, I am getting this error on wmt, any idea on this? thanks 

```
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workdir/seq2seq/xla_spawn.py", line 113, in <module>
    app.run(main, flags_parser=parse_args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/workdir/seq2seq/xla_spawn.py", line 108, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=xla_args.num_cores)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
    start_method=start_method)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 17
```

Comments:
Hi! Usually this error is preceded by another message. The exception with exit code 17 is not the error in itself. Please paste the full error stack trace here, alongside the command you used to launch your script.

Hi Lysandre, here is the full error I get on the google cloud it is
terminated because of this exception

Exception: process 0 terminated with exit code 17

but I do not see any other error messages, do you know what might be
causing this?
thanks for your help.
Best
Rabeeh


"Traceback (most recent call last): File
"/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 193, in
_run_module_as_main "__main__", mod_spec) File
"/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 85, in
_run_code exec(code, run_globals) File "/workdir/seq2seq/xla_spawn.py",
line 79, in <module> app.run(main, flags_parser=parse_args) File
"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py",
line 300, in run _run_main(main, args) File
"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py",
line 251, in _run_main sys.exit(main(argv)) File
"/workdir/seq2seq/xla_spawn.py", line 75, in main xmp.spawn(mod._mp_fn,
args=(), nprocs=xla_args.num_cores) File
"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py",
line 395, in spawn start_method=start_method) File
"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py",
line 157, in start_processes while not context.join(): File
"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py",
line 112, in join (error_index, exitcode) Exception: process 0 terminated
with exit code 17



On Wed, Nov 4, 2020 at 7:45 PM Lysandre Debut <notifications@github.com>
wrote:

> Hi! Usually this error is preceded by another message. The exception with
> exit code 17 is not the error in itself.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8304#issuecomment-721907293>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHH2HI6ICB6HB24S7CWDSOGOMNANCNFSM4TKLTIGA>
> .
>


Hi
thank you for the hint, now I got it, I can debug the rest, thanks for this.
Best
Rabeeh

On Wed, Nov 4, 2020 at 7:50 PM Rabeeh Karimi Mahabadi <rabeeh@google.com>
wrote:

> Hi Lysandre, here is the full error I get on the google cloud it is
> terminated because of this exception
>
> Exception: process 0 terminated with exit code 17
>
> but I do not see any other error messages, do you know what might be
> causing this?
> thanks for your help.
> Best
> Rabeeh
>
>
> "Traceback (most recent call last): File
> "/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 193, in
> _run_module_as_main "__main__", mod_spec) File
> "/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 85, in
> _run_code exec(code, run_globals) File "/workdir/seq2seq/xla_spawn.py",
> line 79, in <module> app.run(main, flags_parser=parse_args) File
> "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py",
> line 300, in run _run_main(main, args) File
> "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py",
> line 251, in _run_main sys.exit(main(argv)) File
> "/workdir/seq2seq/xla_spawn.py", line 75, in main xmp.spawn(mod._mp_fn,
> args=(), nprocs=xla_args.num_cores) File
> "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py",
> line 395, in spawn start_method=start_method) File
> "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py",
> line 157, in start_processes while not context.join(): File
> "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py",
> line 112, in join (error_index, exitcode) Exception: process 0 terminated
> with exit code 17
>
>
>
> On Wed, Nov 4, 2020 at 7:45 PM Lysandre Debut <notifications@github.com>
> wrote:
>
>> Hi! Usually this error is preceded by another message. The exception with
>> exit code 17 is not the error in itself.
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/transformers/issues/8304#issuecomment-721907293>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ARPXHH2HI6ICB6HB24S7CWDSOGOMNANCNFSM4TKLTIGA>
>> .
>>
>