The training wait a long time after load model

chunchun: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f8cfbd298dea60977449a7b92c0f608baf5d145.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f8cfbd298dea60977449a7b92c0f608baf5d145" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f8cfbd298dea60977449a7b92c0f608baf5d145.png" alt="image" data-base62-sha1="4v6Rb74pAPBmAjEAUWDUTUuATRj" width="228" height="500" data-dominant-color="090707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">409×896 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It remains this situation for a long time.<br>
Could you please tell me why?</p>

ptrblck: <p>We would need some more information about your use case to help you out.<br>
What kind of model are you loading?<br>
How long does the loading take?<br>
Are you pushing the model to the GPU or is it still on the CPU?</p>

chunchun: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="63604">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>How long does the loading take?</p>
</blockquote>
</aside>
<p>Thanks a lot for your attention.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/STVIR/pysot">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/STVIR/pysot" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/1d07b7df1bdf964e317676202d8220bbaa1d52fa4804d4cd7413e7720a458861/STVIR/pysot" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/STVIR/pysot" target="_blank" rel="noopener nofollow ugc">GitHub - STVIR/pysot: SenseTime Research platform for single object tracking,...</a></h3>

  <p>SenseTime Research platform for single object tracking, implementing algorithms like SiamRPN and SiamMask. - GitHub - STVIR/pysot: SenseTime Research platform for single object tracking, implementi...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
This is the code that I want to repeat the training process.The model is SiamRPN,one of the trackers in the pysot.The losding takes for many hous and when I press ctrl+c,it shows as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edc7581c3bd97a77685ca4510c242255e15ff89d.png" data-download-href="https://discuss.pytorch.org/uploads/default/edc7581c3bd97a77685ca4510c242255e15ff89d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edc7581c3bd97a77685ca4510c242255e15ff89d.png" alt="image" data-base62-sha1="xVufIcrdCH1eIt4WAfJr85jChyZ" width="690" height="164" data-dominant-color="111111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1021×243 7.54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Although I use ctrl+c,the GPU still shows as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a29bffff635ea24f4a201239b38389034a7d078c.png" alt="image" data-base62-sha1="ncvzSkO1Eoz635xbJhQDb8hH4qU" width="628" height="210"></p>
<p>The gpu-util remains 100%.<br>
The training process is pushed on these three GPUs.And I just stcuk here.<br>
I try some things as changing the num_worker to 0 or decrease the batchsize,both of witch don’t work.<br>
Could you please tell me why?</p>

ptrblck: <p>Is this issue specific to this particular repository and can you execute some of our tutorials or do you see the same error?</p>

chunchun: <p>I think this issue is specific to this repository.<br>
I have seen someone else have the same question in the github.<br>
Could it be a problem with distributed training?I use 3 GPUs instead of 8.<br>
sorry I have no idea.</p>

chunchun: <p>I try single GPU to run,it works.<br>
Maybe the reason of this issue is from distributed_training.<br>
I still haven’t found the real reason.</p>

ptrblck: <p>Are you able to run the <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html">distributed example</a> without any error?</p>

chunchun: <p>I will try it then give a repply.<br>
Thank you again for your attention.</p>

chunchun: <p>I still can’t deal with it.<br>
There is the same situation for the others in my lab.<br>
And we don’t know why.</p>