Exception: process 0 terminated with signal SIGKILL
Author: mobassir94
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

i was using this notebook : https://www.kaggle.com/theoviel/bert-pytorch-huggingface-with-tpu-multiprocessing 

to finetune huggingface’s xlm roberta base model on jigsaw multilingual (ongoing kaggle competition)

this is my first time with torch xla and TPU multiprocessing…!

the code i am trying is exactly this one : https://pastebin.com/fS94MKYc   on a kaggle kernel which gives TPU v3-8

but even for batch_size = 8 i see my jupyter notebook crashes after giving this error message : **Your notebook tried to allocate more memory than is available. It has restarted.**

where i can see other people are using same model with even batch_size = 64

full error message looks like this :


```
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<timed exec> in <module>

/opt/conda/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)
    180         join=join,
    181         daemon=daemon,
--> 182         start_method=start_method)

/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    156 
    157     # Loop on join until it returns True or raises an exception.
--> 158     while not context.join():
    159         pass
    160 

/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    106                 raise Exception(
    107                     "process %d terminated with signal %s" %
--> 108                     (error_index, name)
    109                 )
    110             else:

Exception: process 0 terminated with signal SIGKILL
```
   
same problem is occuring also when i try bert base multilingual of huggingface.
so i am not understanding exactly where in my code i need to make change so that it can work? it seems like the problem is not with the batch size but something else that i am unable to catch.please help,thanks in advance

Comments:
Did you find the solution to the problem?

@jhashekhar  it seems like pytorch xla has  some memory issue itself, xla team is working on it, tf tpu is much better at this moment so i am not using pytorch tpu anymore,probably later this year torch  xla team will solve all the performance issues they  are having at this moment,until then i recommend tf tpu or if you need gpu then pytorch gpu.just my recommendation

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Had this come up when parallel training on GPUS with multiprocessing - thoughts on a solution?

Take a look at this issue: https://github.com/pytorch/xla/issues/1870#issuecomment-612217012

It's pretty long, but they helped me solve this problem last year. I got the model working, but ended up using TF.

> pytorch/xla#1870 (comment)

do you have a summary of what we need to do for solving this for pytorch?

@brando90 please use bf16 and follow this simple tutorial of mine for better understanding : https://www.kaggle.com/mobassir/faster-pytorch-tpu-baseline-for-cld-cv-0-9

Don't forget to reduce batch size,image size etc that fits in xla

I think it will help you to solve your oom error,thanks

> Had this come up when parallel training on GPUS with multiprocessing - thoughts on a solution?

I got this problem. Did you solve it?