Cpu inference - ram increases every iteration

Dan_Erez: <p>The model I’m running causes memory to increase with every iteration.<br>
to load it I do the following:</p>
<pre><code class="lang-auto">def _load_model(model_path):
    model = ModelDef(num_classes=35)
    model.load_state_dict(torch.load(model_path, map_location="cpu"), strict=False)
    model.eval()
    return model
</code></pre>
<p>to run it I do:</p>
<pre><code class="lang-auto">    gc.collect()
    with torch.no_grad():
        input_1_torch = torch.from_numpy(np.float32(input_1)).transpose(1, 2)
        res = _model(input_1_torch)
        output = res.transpose(1, 2).data.numpy() 
</code></pre>
<p>If I remove .eval() things get worse. if I remove torch.no_grad() things get worse.<br>
I also used:</p>
<pre><code class="lang-auto">    for name, param in model.named_parameters():
        param.requires_grad = False
</code></pre>
<p>for extra safety.</p>
<p>I even tried to move to ONNX but I got an error that onnx caffe2 requires an interactive interpreter which I can’t use.</p>
<p>Any help would be very much appreciated.<br>
Thanks Dan</p>

albanD: <p>Hi,</p>
<p>The <code>.data</code> should not be necessary as you’re in a <code>no_grad()</code> block. Same for setting the paremeters not to require gradients.<br>
How do you measure memory usage?<br>
Can you give a bit more context on your code? Does the model save some states by any chance?<br>
Or do you save some outputs for later processing?</p>

Dan_Erez: <p>I removed t".data" and requires_grad = False. No change.<br>
I am measuring memory using ubuntu’s “system monitor” and glances.</p>
<p>I don’t think my code has any states. It has a  couple of static variables other than nn.modules.<br>
The model is mostly sequential convolutions, normalizations, multiplications, sums…etc. Nothing too crazy. It’s not an ensemble or GAN or anything like that.<br>
When training I was using a GPU and it would run iterations including validations for days without issues.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02a133c0a4e1e37f6716a75e81fd59247edf624b.png" data-download-href="https://discuss.pytorch.org/uploads/default/02a133c0a4e1e37f6716a75e81fd59247edf624b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02a133c0a4e1e37f6716a75e81fd59247edf624b_2_690x197.png" alt="image" data-base62-sha1="ngkozgMi6FxE5LHLCnKJwZ2f0f" width="690" height="197" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02a133c0a4e1e37f6716a75e81fd59247edf624b_2_690x197.png, https://discuss.pytorch.org/uploads/default/original/3X/0/2/02a133c0a4e1e37f6716a75e81fd59247edf624b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/02a133c0a4e1e37f6716a75e81fd59247edf624b.png 2x" data-dominant-color="F4F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">848×243 10.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is an example of the first 5 or so iterations. It continues to go up until the process dies from memory issues.</p>

albanD: <p>That is unexpected indeed.<br>
Do you think you can provide a small code sample that reproduces the issue?</p>

Dan_Erez: <p>I can share with you the code by email if you promise to delete it afterwards and not share it with anyone else.<br>
My email is ~~</p>

albanD: <p>It might be better if you can reduce the issue to a smaller code sample before sharing it?</p>
<ul>
<li>Add around your inference loop a print for memory usage change (using something like <a href="https://stackoverflow.com/questions/938733/total-memory-used-by-python-process" rel="nofollow noopener">https://stackoverflow.com/questions/938733/total-memory-used-by-python-process</a> for example).</li>
<li>Make a copy of your code out of your main repo</li>
<li>Replace your actual dataset with a bunch of <code>input = torch.rand(some_hardcoded_size)</code> inside the evaluation loop</li>
<li>Remove your instrumentation code (evaluating the accuracy / plotting etc)</li>
<li>Replace your model with a randomly initialized version (don’t load the state dict)</li>
<li>Replace your model with a single Linear layer that perform the mapping you want (with some views maybe)</li>
<li>Change the input/output sizes so that tests run quickly</li>
</ul>
<p>At each step when you remove something, make sure that the memory behavior is still bad.<br>
Let me know how far you can get. I may be able to look at your code but if it is big, it might take some time to reproduce / pinpoint the problem with the above process as I won’t be familiar with the code.</p>

Dan_Erez: <p>I’ll give those steps a try. I’ll let you know if I got it / if I need more help.<br>
Thanks.</p>

Dan_Erez: <p>Well this is embarrassing but it really seems to not be the fault of pytorch. If I run random inputs with constant size directly through the net it doesn’t increase the ram. I still haven’t found the real issue so I’ll keep this open incase it turns out to be some weird combination of things that include the net.<br>
Thanks!</p>

albanD: <p>Happy to hear that you’re finding new stuff !</p>
<p>So it might be your dataloader? Let me know if you have some of code or torch element you’re not sure about.</p>

Dan_Erez: <p>So after some more debugging I found that if I switch to pytorch cpu the ram stays stable.<br>
So it looks like it was a pytorch related bug after all. None the less I loved your keep it simple stupid debugging methods. I think i’ll hang them up on the wall.</p>
<p>Thanks,<br>
Dan</p>

thiagodma: <p>Hey, I’ve installed pytorch cpu and my RAM keeps increasing on inference. Any clues?</p>
<p>Edit: actually, the problem only exists when I’m in jupyter notebook… If I do inference by running a .py file from terminal it works fine…</p>