Training becomes slower gradually

username: amsword
cooked: <p>The training becomes slower. Anyone has any idea on the issues?</p>
<ol>
<li>
<p>speed vs iteration<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/282fd74d06b7061484ffae667e9154cff32f6468.png" data-download-href="https://discuss.pytorch.org/uploads/default/282fd74d06b7061484ffae667e9154cff32f6468" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/282fd74d06b7061484ffae667e9154cff32f6468_2_690x440.png" alt="image" data-base62-sha1="5JvB3FsQxg5RywVDVqww6dqGgFa" width="690" height="440" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/282fd74d06b7061484ffae667e9154cff32f6468_2_690x440.png, https://discuss.pytorch.org/uploads/default/original/3X/2/8/282fd74d06b7061484ffae667e9154cff32f6468.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/8/282fd74d06b7061484ffae667e9154cff32f6468.png 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">912×582 23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>the grad graph size is consistent. The conclusion is drawn by calling the following function (input argument is loss.grad_fn) to see if there is more node added in the graph. The output here is always 11.</p>
</li>
</ol>
<pre><code class="lang-auto"> def calc_num_node_in_grad_fn(grad_fn):
      result = 0
      if grad_fn is not None:
          result += 1
          if hasattr(grad_fn, 'next_functions'):
              for f in grad_fn.next_functions:
                  result += calc_num_node_in_grad_fn(f)
      return result
</code></pre>
<ol start="3">
<li>
<p>memory cost is not increased. nvidia-smi is called every 30 minutes. The memory size cost is always consistent.</p>
</li>
<li>
<p>pytorch version: 1.4.</p>
</li>
</ol>
<p>Thanks very much</p>

username: amsword
cooked: <p>no forward_hook, backward_hook are called. The network is fater-rcnn on top of the efficient-det’s backbone.</p>

username: amsword
cooked: <p>Checked the time cost for forward, backward, parameter update, data loading, and find the time cost of forwarding becomes large. The network is based on efficient-det’s backbone + faster-rcnn. SyncBN is used here. drop-connect is also applied by masking the output as all-0, which should have no dependency on the old iterations.</p>
<ol>
<li>
<p>forward time<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c23424755296789adf056f1845356ec3c1f08a71.png" data-download-href="https://discuss.pytorch.org/uploads/default/c23424755296789adf056f1845356ec3c1f08a71" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c23424755296789adf056f1845356ec3c1f08a71.png" alt="image" data-base62-sha1="rI0ljRo8ZXHKSTFZRJnw5B2Gfbr" width="690" height="456" data-dominant-color="F9F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">787×521 20.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>backward time<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74327ac2f5fe84cc16bfc2ab7bac07483d528543.png" data-download-href="https://discuss.pytorch.org/uploads/default/74327ac2f5fe84cc16bfc2ab7bac07483d528543" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74327ac2f5fe84cc16bfc2ab7bac07483d528543_2_690x449.png" alt="image" data-base62-sha1="gzVxnVmzdvTQUHBBhu1XEjuxFaX" width="690" height="449" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74327ac2f5fe84cc16bfc2ab7bac07483d528543_2_690x449.png, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74327ac2f5fe84cc16bfc2ab7bac07483d528543.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74327ac2f5fe84cc16bfc2ab7bac07483d528543.png 2x" data-dominant-color="F8F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">786×512 26.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>update time<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9b7d32ac3f7118296e3419a85d53974cf3d82b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9b7d32ac3f7118296e3419a85d53974cf3d82b8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9b7d32ac3f7118296e3419a85d53974cf3d82b8.png" alt="image" data-base62-sha1="xlz5NJVt4SRiY4hjgli7VbfOh2U" width="690" height="434" data-dominant-color="F9F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">757×477 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>data loading time<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bb96772fecdc2214160a5d45942fdfd70da93e.png" data-download-href="https://discuss.pytorch.org/uploads/default/09bb96772fecdc2214160a5d45942fdfd70da93e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09bb96772fecdc2214160a5d45942fdfd70da93e_2_690x424.png" alt="image" data-base62-sha1="1o6cx3xfyhVZkOLIJIzq4C7lS5E" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09bb96772fecdc2214160a5d45942fdfd70da93e_2_690x424.png, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bb96772fecdc2214160a5d45942fdfd70da93e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bb96772fecdc2214160a5d45942fdfd70da93e.png 2x" data-dominant-color="F9F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">776×477 21.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ol>

username: ptrblck
cooked: <p>Is the forward or loss calculation object-dependent, i.e. is the computation increased for each new candidate?<br>
I’m not familiar with your code, but could the model output more candidates, which would have to be filtered out during the forward pass? The backward, <code>update_time</code>, and data loading might be constant + noise.</p>

username: amsword
cooked: <p>Thanks for your reply. It is faster-rcnn and retinanet as well. The computations theoretically are consistent for different iterations.</p>

username: ptrblck
cooked: <p>How did you profile the different parts of the code?<br>
Could you post the code snippet used for the profiling, please?</p>

username: amsword
cooked: <p>just to use time.time() before model(feature) and then use time.time() to capture the time elapsed. Then, print out.</p>

username: ptrblck
cooked: <p>If you are using the GPU, you would need to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>, since CUDA operations are executed asynchronously.<br>
If you don’t manually synchronize, the next blocking operation will accumulate the timings from the previous ops, so that your results might give you the wrong bottlenecks.</p>