Newbie questions - torch on Google Colab

username: CarloCattano
cooked: <p>Im training a RAVE model and having some questions.<br>
Its my first time using the library and ml training ever, so forgive my ignorance.</p>
<p>Got a colab pro plan just to avoid disconnections but still getting quite some.<br>
Googling around about batch size &amp; num_workers yield few results, but still I dont get it fully.</p>
<ul>
<li>
<ol>
<li>The first warning Im worried about is this one. Apparently colab complains about the number of workers. Note that Im running the python script via !python train_rave.py:</li>
</ol>
</li>
</ul>
<pre><code class="lang-auto">UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
</code></pre>
<ul>
<li>
<ol start="2">
<li>Something odd is happening with my ckpt’s. They seem to load fine but I get the following warning once I resume a training after a disconnect:</li>
</ol>
</li>
</ul>
<pre><code class="lang-auto">model_checkpoint.py:343: UserWarning: The dirpath has changed from 'runs/bulgarianvoices/rave/version_0/checkpoints' to 'runs/bulgarianvoices/rave/version_1/checkpoints', therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.
  f"The dirpath has changed from {dirpath_from_ckpt!r} to {self.dirpath!r},
</code></pre>
<p>Just want to check with experienced users if this is just normal or something I should worry about. RAVE trains on audio with 8 Batches and 8 workers.</p>
<p>Regarding google drive I also have some doubts.</p>
<ul>
<li>
<ol start="3">
<li>Are previous ckpt’s at all necessary ?<br>
could I just remove previous runs and just keep the last.ckpt and corresponding run ? My google drive is filling up quite fast otherwise<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b3afe0b5d35adf4ed8bb3abba0cb6cf503d2d15.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b3afe0b5d35adf4ed8bb3abba0cb6cf503d2d15" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b3afe0b5d35adf4ed8bb3abba0cb6cf503d2d15.png" alt="image" data-base62-sha1="aJwcs0mK8RsYalLUdKBoMqG5kTX" width="331" height="500" data-dominant-color="404040"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">402×607 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ol>
</li>
</ul>
<p>After training for a while I also get a “not available” message on the connect colab button , where normally displays RAM and DISK usage. The script still keeps working but is just interesting to me, with a pro account still don’t have much reliability on that sense.</p>
<p>I would like to still use gdrive to keep the ckpt’s , otherwise on every disconnect I would loose all the work. But I guess is slowing the process to have all running from a gdrive instance.</p>
<p>Any tips or references to docs in this realms will be very appretiated. I dont have experience with torch but Im confortable with python/system etc.</p>
<p>This is the project I run in colab <a href="https://github.com/acids-ircam/RAVE" rel="noopener nofollow ugc">https://github.com/acids-ircam/RAVE</a></p>
<pre><code class="lang-auto">tqdm==4.62.3
effortless-config==0.7.0
einops==0.4.0
librosa==0.8.1
matplotlib==3.5.1
numpy==1.20.3
pytorch-lightning==1.6.1
scikit_learn==1.0.2
scipy==1.7.3
soundfile==0.10.3.post1
termcolor==1.1.0
torch==1.11.0
tensorboard==2.8.0
GPUtil==1.4.0
git+https://github.com/caillonantoine/cached_conv.git@v2.3.5#egg=cached_conv
git+https://github.com/caillonantoine/UDLS.git#egg=udls
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b3a2a27a5a1f2b35f2e756b73b07b6cb9c5ab49.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b3a2a27a5a1f2b35f2e756b73b07b6cb9c5ab49" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b3a2a27a5a1f2b35f2e756b73b07b6cb9c5ab49.png" alt="image" data-base62-sha1="m9cCL3xn3NYMroc1XVsSEgkfjlv" width="690" height="466" data-dominant-color="434343"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">968×655 31.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Its really hard to see the effects on changes made to the code since the waiting period is really long , so I would love to optimize it with colab specific practices. Thinking on running the code in cells instead of from !python</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="CarloCattano" data-post="1" data-topic="151656">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carlocattano/48/48464_2.png" class="avatar"> CarloCattano:</div>
<blockquote>
<p>The first warning Im worried about is this one. Apparently colab complains about the number of workers.</p>
</blockquote>
</aside>
<p>This warning is raised from the <code>DataLoader</code> as it tries to suggest the recommended number of workers based on the system resources <a href="https://github.com/pytorch/pytorch/blob/fb0f285638338da93960d2b654a59c9639671fc0/torch/utils/data/dataloader.py#L478-L505">here</a>.<br>
Try to lower the number to the suggested and and check if it yields a speedup.</p>
<aside class="quote no-group" data-username="CarloCattano" data-post="1" data-topic="151656">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carlocattano/48/48464_2.png" class="avatar"> CarloCattano:</div>
<blockquote>
<p>Something odd is happening with my ckpt’s. They seem to load fine but I get the following warning once I resume a training after a disconnect:</p>
</blockquote>
</aside>
<p>This warning seems to be raised from Lightning <a href="https://github.com/PyTorchLightning/pytorch-lightning/blob/fab2ff35ad98ed021ee1d151bb67d3a4088c848b/pytorch_lightning/callbacks/model_checkpoint.py#L341-L346">here</a> and I don’t know what might be causing it. Based on the message I guess different “versions” are stored and the latest one is used to reload the model?</p>
<aside class="quote no-group" data-username="CarloCattano" data-post="1" data-topic="151656">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carlocattano/48/48464_2.png" class="avatar"> CarloCattano:</div>
<blockquote>
<p>Are previous ckpt’s at all necessary ?<br>
could I just remove previous runs and just keep the last.ckpt and corresponding run ? My google drive is filling up quite fast otherwise</p>
</blockquote>
</aside>
<p>Could you explain what “version” refers to? If these are corresponding to e.g. the epoch, you could try to save only the last or best epoch to avoid the storage issue.</p>

username: CarloCattano
cooked: <p>Thanks for the input !</p>
<blockquote>
<p>This warning seems to be raised from Lightning <a href="https://github.com/PyTorchLightning/pytorch-lightning/blob/fab2ff35ad98ed021ee1d151bb67d3a4088c848b/pytorch_lightning/callbacks/model_checkpoint.py#L341-L346" rel="noopener nofollow ugc">here</a> and I don’t know what might be causing it. Based on the message I guess different “versions” are stored and the latest one is used to reload the model?</p>
</blockquote>
<p>Yes, and sometimes because of how colab handles disconnect I guess I also get this one</p>
<pre><code class="lang-auto">UserWarning: You're resuming from a checkpoint that ended before the epoch ended. This can cause unreliable results if further training is done. Consider using an end-of-epoch checkpoint or enabling fault-tolerant training: https://pytorch-lightning.readthedocs.io/en/stable/advanced/fault_tolerant_training.html
  "You're resuming from a checkpoint that ended before the epoch ended. This can cause unreliable"
</code></pre>
<p>Then Im again unsure if acting correctly</p>
<blockquote>
<p>This warning is raised from the <code>DataLoader</code> as it tries to suggest the recommended number of workers based on the system resources <a href="https://github.com/pytorch/pytorch/blob/fb0f285638338da93960d2b654a59c9639671fc0/torch/utils/data/dataloader.py#L478-L505" rel="noopener nofollow ugc">here</a>.<br>
Try to lower the number to the suggested and and check if it yields a speedup.</p>
</blockquote>
<p>I tried with my own fork of the project to change Batches and workers and didnt turn out so well, but needs more testing</p>
<blockquote>
<p>Could you explain what “version” refers to? If these are corresponding to e.g. the epoch, you could try to save only the last or best epoch to avoid the storage issue.</p>
</blockquote>
<p>Thats exactly what I was thinking I could do and thanks to your comment now I know that I can do it without fear of loosing anything in the final export to .ts<br>
My fear was that deleting previous chkpt’s would somehow affect the final result when exporting to ts.<br>
I would love to have a good GPU and ditch colab for good , but at the moment is all I can work with to experiment and learn .</p>
<p>Will look for resources also on understanding tensorboard better since I dont really have a clue of what Im doing and all looks so cryptic.</p>
<p>Getting around 2 Million steps at the moment , with the mentioned limitations and perks , but at least getting somewhere I hope .</p>
<p>Thanks again for the input <a class="mention" href="/u/ptrblck">@ptrblck</a></p>