Memory leak during inference

abc_abc: <p>I am trying <a href="https://github.com/williamyang1991/VToonify" rel="noopener nofollow ugc">Vtoonify</a> model but in inference the memory keep increasing after every couple of images. In order to lower the consumption I set a standard size for input image like 320*320 so all tensors are of same size but still it keep consuming the memory. I tried dynamic quantization but results are very bad . Here is my code for quantization:</p>
<pre><code class="lang-auto">MODEL_REPO = 'PKUWilliamYang/VToonify'

vtoonify = VToonify(backbone='dualstylegan')

vtoonify.load_state_dict(torch.load(huggingface_hub.hf_hub_download(MODEL_REPO,
                                                                    'models/vtoonify_d_cartoon/vtoonify_s026_d0.5.pt'),
                                    map_location=lambda storage, loc: storage)["g_ema"], strict=False)

vtoonify.eval()

quantized_model = torch.quantization.quantize_dynamic(vtoonify, {nn.Linear}, dtype=torch.qint8)

quantized_model.qconfig = torch.ao.quantization.get_default_qconfig('x86')

requires_grad(quantized_model.generator, False)
requires_grad(quantized_model.res, False)

torch.save(
    {
        #"g": g_module.state_dict(),
        #"d": d_module.state_dict(),
        "g_ema": quantized_model.state_dict(),
    },
    './converted_dynamic_vtoonify_s026_d0.5.pt'
)
</code></pre>
<p>This should be the result<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/397a7d4cfdcff725434480af5825a52b8d2cdf16.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/397a7d4cfdcff725434480af5825a52b8d2cdf16" title="result1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/397a7d4cfdcff725434480af5825a52b8d2cdf16_2_609x500.jpeg" alt="result1" data-base62-sha1="8ctEbVFeO8fk9eYlRJNhODYOlTw" width="609" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/397a7d4cfdcff725434480af5825a52b8d2cdf16_2_609x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/397a7d4cfdcff725434480af5825a52b8d2cdf16_2_913x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/397a7d4cfdcff725434480af5825a52b8d2cdf16_2_1218x1000.jpeg 2x" data-dominant-color="8E867F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">result1</span><span class="informations">1600×1312 89.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But after quantization I get this output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cb1c26d745227ce6fb45aadcd3a1983727af0336.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cb1c26d745227ce6fb45aadcd3a1983727af0336" title="result"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1c26d745227ce6fb45aadcd3a1983727af0336_2_662x500.jpeg" alt="result" data-base62-sha1="sYNeXvFyj9GZjuQEcYrb1ZR4g6O" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1c26d745227ce6fb45aadcd3a1983727af0336_2_662x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1c26d745227ce6fb45aadcd3a1983727af0336_2_993x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1c26d745227ce6fb45aadcd3a1983727af0336_2_1324x1000.jpeg 2x" data-dominant-color="6F6458"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">result</span><span class="informations">1568×1184 69.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried mixed precision and also using <code>with torch.no_grad()</code> but got no luck.</p>
<p>I started with 4GB GPU then moved to 8gb and now 12gb still it manages to eat all of the memory.</p>