How can I release the unused gpu memory?

username: yf_zhang
cooked: <p>I tried to del unused variable and use ‘torch.cuda.empty_cache()’ to release the gpu memory. Bute I found the used gpu memory is constantly changing but the maximum value is unchanged.<br>
I build the resnet18 in my own way, but the used gpu memory is obviously larger than the official implementation in torch.vision. So how can I find the reason?<br>
Thanks for attention!</p>
<pre><code class="lang-auto">    def forward(self, bottoms):
        # string -&gt; feature_map list
        feature_pool = dict()
        bottoms = bottoms if isinstance(bottoms, list) else [bottoms]
        if (self.get_device_id() &gt;= 0):
            for idx, bottom in enumerate(bottoms):
                bottoms[idx] = bottoms[idx].cuda(device=self.get_device_id())

        for id, i_idx in enumerate(self.input_idx):
            feature_pool["bottom_{}".format(id)] = [bottoms[i_idx]]
        del bottoms
        torch.cuda.empty_cache()

        for idx, dag_node in enumerate(self.dag_list):
            local_bottoms = []
            for x in dag_node.bottoms:
                local_bottoms.extend(feature_pool[x])

                is_x_depended = any([True if x in dag.bottoms else False for dag in self.dag_list[(idx+1):]])
                if (not is_x_depended) and (x not in self.top_names):
                    del feature_pool[x]
                    torch.cuda.empty_cache()
            local_tops = self._modules[dag_node.scope].forward(local_bottoms)

            assert len(dag_node.tops) == len(local_tops)
            for i in range(len(dag_node.tops)):
                feature_pool[dag_node.tops[i]] = [local_tops[i]]
            del local_tops, local_bottoms
            torch.cuda.empty_cache()

        feature_list = []
        for name in self.top_names:
            feature_list.extend(feature_pool[name])
            del feature_pool[name];torch.cuda.empty_cache()

        return feature_list
</code></pre>

username: ptrblck
cooked: <p>To release the memory, you would have to make sure that all references to the tensor are deleted and call <code>torch.cuda.empty_cache()</code> afterwards.<br>
E.g. <code>del bottoms</code> should only delete the internal <code>bottoms</code> tensor, while the global one should still be alive.</p>
<p>Also, note that <code>torch.cuda.empty_cache()</code> will not avoid out of memory issues, since the cache is reused, not lost.</p>

username: yf_zhang
cooked: <p>I del local_bottoms after using it as the input of forward function.</p>

username: yf_zhang
cooked: <p>Can you please read my code and point out the reasons for the large ratio of gpu memory? I spend a lot of time but it doesn’t work.</p>

username: ptrblck
cooked: <p>Removing the local reference will not delete the global tensor.<br>
If you cannot free the cache, then a reference is still pointing to the tensor as shown here:</p>
<pre><code class="lang-python">def fun(tensor):
    print(torch.cuda.memory_allocated() / 1024**2)
    # Delete local reference
    del tensor
    print(torch.cuda.memory_allocated() / 1024**2)
    return

# Check that memory is empty
print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_cached())
&gt; 0

# Create tensor
x = torch.randn(1024 * 1024, device='cuda')
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 4.0
print(torch.cuda.memory_cached() / 1024**2)
&gt; 20.0

# Call fun and check, if x is still alive
fun(x)
&gt; 4.0
&gt; 4.0

print(x.device) # still alive
&gt; cuda:0
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 4.0
print(torch.cuda.memory_cached() / 1024**2)
&gt; 20.0

# Delete global tensor
del x
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 0.0
print(torch.cuda.memory_cached() / 1024**2)
&gt; 20.0

# Now empty cache
torch.cuda.empty_cache()
print(torch.cuda.memory_cached() / 1024**2)
&gt; 0.0
</code></pre>

username: yf_zhang
cooked: <p>Thanks for replying. I konw your explanation. But I think “del bottoms” not working is not the reason for hight ratio of gpu usage memory.</p>

username: ptrblck
cooked: <p>If I understand your issue correctly you are trying to empty the cache, which doesn’t seem to be working, right?<br>
If that’s the case, you would have to delete all references to the tensors you would like to delete so that the cache can be emptied.</p>
<p>If I misunderstood it, please correct me.</p>

username: yf_zhang
cooked: <p>Yes, I tried to avoid using temporary variables and delete unusable variable.In forward function of each module, I delete other feature_map tensor before return result.</p>

username: ptrblck
cooked: <p>Did you check, if the cached memory was decreased using this approach?</p>

username: yf_zhang
cooked: <p>If I delete tensor and use empty_cache, the used gpu memory is constantly changed. But the maximum ratio of used gpu memory is not changed.</p>

username: ptrblck
cooked: <p>You won’t avoid the max. memory usage by removing the cache.<br>
As explained before, <code>torch.cuda.empy_cache()</code> will only release the cache, so that PyTorch will have to reallocate the necessary memory and might slow down your code<br>
The memory usage will be the same, i.e. if your training has a peak memory usage of 12GB, it will stay at this value.<br>
You will only temporarily reduce the allocated memory, which will then be reallocated if necessary.</p>

username: yf_zhang
cooked: <p>I agree your opinion. But when I define the cnn model by code showed in question. The peak memory usage is 4 times larger than offfical resnet model in torchvision.</p>

username: ptrblck
cooked: <p>I would recommend to add debug statements using <code>print(torch.cuda.max_memory_allocated())</code> to try to narrow down which operations are wasting the memory.</p>
<p>Just by skimming through the code, it seems that some lists and dicts are temporarily used and freed later. This might increase the peak memory, e.g. if you are storing the complete feature maps first and delete them one by one later.</p>

username: yf_zhang
cooked: <p>When I delete tensor and use empty_cache, the memory usage will decrease only when one-batch train process done rather than where I use “torch.cuda.empy_cache()”.</p>

username: yf_zhang
cooked: <p>Dear ptrblck, I found the gpu memory decrease from 5390M to 256M if I use empy_cache. The decease happens when model infer the validation dataset.</p>

username: ptrblck
cooked: <p>That might be expected and PyTorch will reallocate the memory, if needed.<br>
You can clear the cache, but won’t be able to reduce the peak memory, and might just slow down the code using it.</p>
<p>If your custom ResNet implementation uses more memory than the <code>torchvision</code> implementation, I would still recommend to compare both implementations by adding the mentioned print statements and narrow down which part of your code uses more memory.</p>