Why time steps is increasing?

Rafi: <p>I am training a pytorch model and I remark that the time steps is increasing as mentioned below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d0a55921fc437c644d313bd41cca0387321dbd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/2d0a55921fc437c644d313bd41cca0387321dbd3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d0a55921fc437c644d313bd41cca0387321dbd3.png" alt="image" data-base62-sha1="6qrCT2AVz69xCj8bK1HReucdlbZ" width="285" height="500" data-dominant-color="E0F2E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">473×828 36.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I remark as well that my GPU is 0% use<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3ddd9f57e659b9cb3a558ac3bc170e70dbba5da.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3ddd9f57e659b9cb3a558ac3bc170e70dbba5da" title="gifgit"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3ddd9f57e659b9cb3a558ac3bc170e70dbba5da_2_690x455.png" alt="gifgit" data-base62-sha1="nnD8OjeDEFZDzxA7BA6wCeQt6zM" width="690" height="455" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3ddd9f57e659b9cb3a558ac3bc170e70dbba5da_2_690x455.png, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3ddd9f57e659b9cb3a558ac3bc170e70dbba5da.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3ddd9f57e659b9cb3a558ac3bc170e70dbba5da.png 2x" data-dominant-color="3F1B33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gifgit</span><span class="informations">890×587 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the transform that I am using:</p>
<pre><code class="lang-auto">def get_transform(config):
    scale = torchvision.transforms.Lambda(lambda x: x.astype(np.float32) / 255.)
    identity = torchvision.transforms.Lambda(lambda x: x)
    size = config.transform.mpiifacegaze_face_size
    if size != 448:
        resize = torchvision.transforms.Resize((size, size))
    else:
        resize = identity
    if config.transform.mpiifacegaze_gray:
        to_gray = torchvision.transforms.Lambda(lambda x: cv2.cvtColor(
            cv2.equalizeHist(cv2.cvtColor(x, cv2.COLOR_BGR2GRAY)), cv2.
            COLOR_GRAY2BGR))
    else:
        to_gray = identity

    transform = torchvision.transforms.Compose([
        resize,
        to_gray,
        torchvision.transforms.Lambda(lambda x: np.transpose(x, [2, 0, 1])),
        scale,
        torch.from_numpy,
        torchvision.transforms.Normalize(mean=[0.406, 0.456, 0.485],
                                         std=[0.225, 0.224, 0.229]),
    ])
    return transform
</code></pre>

ptrblck: <p>Are you only applying the posted transformation in your code snippet without any model training?<br>
If so, then I would expect the GPU utilization to be at 0%, since the CPU will be used.<br>
For the increase in iteration time: I guess you might be storing some tensors, which are still attached to the computation graph, and are indeed training the model. This would increase the memory usage as well as the processing time.</p>

Rafi: <p>No, I am training a deep model based on mobilenet.</p>