How to disable model's gpu cache?

username: Klink
cooked: <p>I use these options for model.eval: non_blocking = True, use_cache=False.</p>
<p>But my model use GPU cache on every response.</p>
<p>I know torch.cuda.empty_cache(); can delete the model caches.</p>
<p>But caching sequence make delay on response time.</p>
<p>It was not do caching before last train.</p>
<p>But after gradient training, It use more GPU memory every response. (More then last response, Before using torch.cuda.empty_cache()<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>How can I disable model’s caching?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Klink" data-post="1" data-topic="148104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ea666f/48.png" class="avatar"> Klink:</div>
<blockquote>
<p>I use these options for model.eval: non_blocking = True, use_cache=False.</p>
</blockquote>
</aside>
<p>Could you describe where <code>use_cache=False</code> is used?</p>
<aside class="quote no-group" data-username="Klink" data-post="1" data-topic="148104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ea666f/48.png" class="avatar"> Klink:</div>
<blockquote>
<p>But caching sequence make delay on response time.</p>
</blockquote>
</aside>
<p>Could you also explain how the caching memory allocator is delaying the response?<br>
Being able to reuse already allocated memory from the cache will speed up the performance of your script as the synchronizing <code>cudaMalloc</code> calls would be avoided.<br>
<code>torch.cuda.empty_cache()</code> will also synchronize your code and show most likely a performance drop.</p>
<p>If you want to globally disable the caching mechanism, use <code>export PYTORCH_NO_CUDA_MEMORY_CACHING=1</code> which is used as a debug flag as it wouldn’t make sense to use it in “production” code.</p>

username: Klink
cooked: <p>I tested it on config.json, from_pretrained argument and model.generate argument.<br>
But every responses make memory leak on my GPU.<br>
Model’s memory usage is bigger then last response everytime.<br>
And response delay is longer then last response time.</p>
<p>I think it is GPU caching, because additional memory usage can delete by torch.cuda.empty_cache().<br>
Can I using PYTORCH_NO_CUDA_MEMORY_CACHING value?</p>

username: ptrblck
cooked: <p>The caching allocator does not cause the memory “leak” and I guess you are storing data on the GPU unnecessarily.</p>

username: Klink
cooked: <p>All environments are the same as when it work normally.<br>
Only the weights file is different from before.<br>
Could deepspeed training makes these issues on finished file?</p>

username: ptrblck
cooked: <p>I don’t know if Deepspeed or any other part of your code is responsible for the increased memory usage.<br>
Usually tensors, which are still attached to the entire computation graph, are stored in e.g. a <code>list</code> which increases the memory usage and might look like a leak (but is in fact expected behavior in this case).<br>
You can search this forum for similar issues and check how they were solved.</p>
<p>With that being said, you can certainly disable the cache and see if if helps. I would claim you would see the same increasing memory usage and would additionally slow down your code.</p>

username: Klink
cooked: <p>I think… I need more time to find any solutions.<br>
Thx. Have a nice day.</p>