Training time increasing with steps

Nemfor: <p>Hello! I have the next problem: my torch model slows down by the end of an epoch and starts to perform well in a new epoch.<br>
So, I use tqdm to measure iter/second performance and have the next picture: at start of training performance is about 20 iter/sec, but it slows down with increasing iteration number and finished with about 4it/sec. In new epoch it start again from 20 and return to 4 by the end. I remove all from training loop, now that is like in pytorch tutorial, no any additional steps.<br>
Use Dataloader with num_workes&gt;1 + pin_memory, I have about 250 GB of data, which pass to Loader <strong>init</strong>, so my Dataset just returns index, no any loading data from disk and no transformations, all data stored in init in the way it passed to forward.<br>
How can i find this bottleneck of my model?</p>

ptrblck: <p>Could you check if you are appending some tensors to a list and could thus be storing the computation graphs throughout the epoch? This should be visible in an increased memory usage in the first epoch. Also, are you using <code>backward(retain_graph=True)</code> or any other “special” setup?</p>

Nemfor: <p>Thanks for the answer!<br>
I haven’t seen anything strange in the training loop:</p>
<pre><code>    with tqdm(total=len(train_generator)) as prbar:
        for batch_idx, batch in enumerate(train_generator, 1):
            output = self._model(batch[:3]).squeeze(dim=1)
            target = batch[3].to(self._device)
            loss = loss_func(output, target)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

            total_loss += float(loss.item()) * target.size(0)
            y_true.extend(list(target.data.cpu().numpy()))
            y_pred.extend(list(output.data.cpu().numpy()))
            l1_part += float(l1_weight * weights_norm_1 / float(loss) * 100)
            l2_part += float(l2_weight * weights_norm_2 / float(loss) * 100)
</code></pre>
<p>Can any things be in the impelemnted model class that can slows the perofmance?</p>

ptrblck: <p>Where are these values or tensors coming from?</p>
<pre data-code-wrap="python"><code class="lang-python">            l1_part += float(l1_weight * weights_norm_1 / float(loss) * 100)
            l2_part += float(l2_weight * weights_norm_2 / float(loss) * 100)
</code></pre>
<p>Could you check if some of them are attached to a computation graph and would thus have a valid <code>.grad_fn</code>?<br>
If so, <code>detach()</code> them before accumulating.</p>
<aside class="quote no-group" data-username="Nemfor" data-post="3" data-topic="136586">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nemfor/48/33224_2.png" class="avatar"> Nemfor:</div>
<blockquote>
<p>Can any things be in the impelemnted model class that can slows the perofmance?</p>
</blockquote>
</aside>
<p>Yes, if your model increases the workload in each iteration e.g. by using longer sequences or by backpropagating through all previous iterations, you would see a slowdown.</p>

Nemfor: <p>This parts is just additional regularization + sample weights:<br>
if use_weights:<br>
weights = batch[4].to(self._device)<br>
loss = (loss * weights).mean()<br>
if l1_weight:<br>
weights_norm_1 = 0<br>
for i, params in enumerate(self._model.parameters(), 0):<br>
if i != 0:  # not embedding layer<br>
weights_norm_1 += torch.norm(params, 1)<br>
loss = loss + l1_weight * weights_norm_1<br>
Where can I use .grad_fn to understand if they have childs?</p>
<p>List item</p>

ptrblck: <p>You can print it directly as it’s a tensor attribute:</p>
<pre><code class="lang-python">print(loss.grad_fn)
print(l1_weight.grad_fn)
...
</code></pre>

Nemfor: <p>Thanks, will try to do this.<br>
We have two similar models (torch/keras) and now keras performs two times faster (keras ~ 23 min per epoch/ torch ~ 40-50 min) that looks a bit strange.</p>