Exception: process 0 terminated with signal SIGKILL

mobassir94: <p>i was using this notebook : <a href="https://www.kaggle.com/theoviel/bert-pytorch-huggingface-with-tpu-multiprocessing" rel="nofollow noopener">https://www.kaggle.com/theoviel/bert-pytorch-huggingface-with-tpu-multiprocessing</a></p>
<p>to finetune huggingface’s xlm roberta base model on jigsaw multilingual (ongoing kaggle competition)</p>
<p>this is my first time with torch xla and TPU multiprocessing…!</p>
<p>the code i am trying is exactly this one : <a href="https://pastebin.com/fS94MKYc" rel="nofollow noopener">https://pastebin.com/fS94MKYc</a> on kaggle kernel which gives TPU v3-8</p>
<p>but even for batch_size = 8 i see my jupyter notebook crashes after giving this error message : <strong>Your notebook tried to allocate more memory than is available. It has restarted.</strong></p>
<p>where i can see other people are using same model with even batch_size = 64</p>
<p>full error message looks like this :</p>
<ol>
<li>
<hr>
</li>
<li>
<p>Exception                                 Traceback (most recent call last)</p>
</li>
<li>
<p> in </p>
</li>
</ol>
<ul>
<li>/opt/conda/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)</li>
</ul>
<ol start="4">
<li>
<p>180         join=join,</p>
</li>
<li>
<p>181         daemon=daemon,</p>
</li>
<li>
<p>–&gt; 182         start_method=start_method)</p>
</li>
</ol>
<ul>
<li>/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)</li>
</ul>
<ol start="7">
<li>
<p>156</p>
</li>
<li>
<p>157     # Loop on join until it returns True or raises an exception.</p>
</li>
<li>
<p>–&gt; 158     while not context.join():</p>
</li>
<li>
<p>159         pass</p>
</li>
<li>
<p>160</p>
</li>
</ol>
<ul>
<li>/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)</li>
</ul>
<ol start="12">
<li>
<p>106                 raise Exception(</p>
</li>
<li>
<p>107                     “process %d terminated with signal %s” %</p>
</li>
<li>
<p>–&gt; 108                     (error_index, name)</p>
</li>
<li>
<p>109                 )</p>
</li>
<li>
<p>110             else:</p>
</li>
</ol>
<ul>
<li>Exception: process 0 terminated with signal SIGKILL</li>
</ul>
<p>so i am not understanding exactly where in my code i need to make change so that it can work? it seems like the problem is not with the batch size but something else that i am unable to catch.please help,thanks in advance</p>

vfdev-5: <p>Hi <a class="mention" href="/u/mobassir94">@mobassir94</a>, you put category “Ignite”, but it seems not to be related to PyTorch-Ignite, right ?</p>

mobassir94: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a>  sorry i didn’t find xla there in category</p>

vfdev-5: <p>No problems, maybe we can ask <a class="mention" href="/u/smth">@smth</a> for that.</p>

machouz: <p><a class="mention" href="/u/mobassir94">@mobassir94</a> can you check if this work<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/PyTorchLightning/pytorch-lightning/issues/1590#issuecomment-619661785" target="_blank" rel="nofollow noopener">github.com/PyTorchLightning/pytorch-lightning</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/PyTorchLightning/pytorch-lightning/issues/1590#issuecomment-619661785" target="_blank" rel="nofollow noopener">Colab TPU : process terminated with signal SIGKILL</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-24" data-time="04:23:40" data-timezone="UTC">04:23AM - 24 Apr 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-04-27" data-time="01:31:04" data-timezone="UTC">01:31AM - 27 Apr 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/Colanim" target="_blank" rel="nofollow noopener">
          <img alt="Colanim" src="https://avatars2.githubusercontent.com/u/43774355?v=4" class="onebox-avatar-inline" width="20" height="20">
          Colanim
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
I'm trying to train BART (with transformers library) on Colab TPU. I followed the TPU documentation of Pytorch Lightning, but...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">bug / fix</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">help wanted</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Brando_Miranda: <p>did you ever figure out a) what was causing your issue and b) how you solved it?</p>
<p>Thanks in advance!</p>

mobassir94: <p><a class="mention" href="/u/brando_miranda">@Brando_Miranda</a>  yes, it’s OOM issue,i reduce batch size,recent version of torch xla is much better</p>