Libtorch inference causes CUDA out of memory

c0d3r43v3r: <p>Hi there,</p>
<p>I have successfully transformed a very complex pytorch Python model into C++ libtorch and it wasn’t easy.</p>
<p>The input to the model are 2 grayscale image tensors.<br>
The output are 3 tensors.</p>
<p>I was able to run inference in C++ and get the same results as the pytorch inference.<br>
BUT running inference on several images in a row causes CUDA out of memory:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 5.78 GiB total capacity; 3.54 GiB already allocated; 21.62 MiB free; 3.81 GiB reserved in total by PyTorch)<br>
If reserved memory is &gt;&gt; allocated memory<br>
try setting max_split_size_mb to avoid fragmentation.<br>
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>The inference code looks like this:</p>
<pre><code class="lang-auto">
torch::jit::script::Module model = torch::jit::load(
                                     "PATH TO MODEL", torch::kCUDA);

torch::jit::getProfilingMode() = false;

for (int i = 0;i &lt; 100;++i)
{
    // The image paths are different on every loop iteration !
    cv::Mat img0_raw = cv::imread("PATH TO IMAGE", cv::IMREAD_GRAYSCALE);
    cv::Mat img1_raw = cv::imread("PATH TO IMAGE", cv::IMREAD_GRAYSCALE);
    
    cv::resize(img0_raw, img0_raw, cv::Size(480, 480), 0, 0, cv::INTER_LINEAR);
    cv::resize(img1_raw, img1_raw, cv::Size(480, 480), 0, 0, cv::INTER_LINEAR);
    
    cv::Mat img0Infer_Raw, img1Infer_Raw;
    img0_raw.convertTo(img0Infer_Raw, CV_32FC1, 1/255.0);
    img1_raw.convertTo(img1Infer_Raw, CV_32FC1, 1/255.0);
  
    torch::Tensor img0Tensor = torch::from_blob(img0Infer_Raw.data, { 1, 480, 480 }, 
                                 at::kFloat).unsqueeze(0).to(torch::kCUDA);
  
    torch::Tensor img1Tensor = torch::from_blob(img1Infer_Raw.data, { 1, 480, 480 }, 
                                 at::kFloat).unsqueeze(0).to(torch::kCUDA);
  
    std::vector&lt;torch::jit::IValue&gt; inputs {img0Tensor, img1Tensor};
    auto outputs = model.forward(inputs).toTuple();
    torch::Tensor mconf = outputs-&gt;elements()[0].toTensor();
    torch::Tensor kpts0_c = outputs-&gt;elements()[1].toTensor();
    torch::Tensor kpts1_c = outputs-&gt;elements()[2].toTensor();
      
    torch::Tensor cmconf = mconf.to(torch::kCPU);
    torch::Tensor ckpts0_c = kpts0_c.to(torch::kCPU);
    torch::Tensor ckpts1_c = kpts1_c.to(torch::kCPU);
}

</code></pre>
<p>Some observations:</p>
<ol>
<li>3.54 already allocated by pytorch</li>
<li>3.81 is reserved by pytorch</li>
<li>if the images are the same for every iteration then it doesn’t crash !</li>
<li>i’ve tried calling  c10::cuda::CUDACachingAllocator::emptyCache();  but it didin’t solve the problem</li>
</ol>
<p>I suspect that pytorch cache or the reserved memory is too small,<br>
Any help will be appreciated<br>
Thanks in advance</p>
<p>Omer</p>

eqy: <p>One thing I would check (not entirely familiar with this use-case) is if there are some model activations being inadvertently stored.</p>
<p>Does calling</p>
<pre><code class="lang-auto">model.requires_grad_(false);
model.train(false);
</code></pre>
<p>before running inference change the behavior at all?</p>

c0d3r43v3r: <p>Thanks for idea but doing this doesn’t change the out of memory problem…<br>
And i couldn’t find any requires_grad_(false)  so i’ve used  torch::requires_grad(false) which made no difference.</p>

eqy: <p>Based on the original error message, it seems that a significant portion of GPU memory isn’t used by PyTorch, is there there some other activity on the same GPU while you are running the code?</p>
<p>I would also check to see if the original Python implementation has the same behavior, and you could check the memory utilization at each step e.g., with <a href="https://pytorch.org/docs/stable/generated/torch.cuda.memory_summary.html" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.memory_summary — PyTorch 1.12 documentation</a>. I would expect the pattern of allocations/deallocations on the GPU to be basically the same in Python vs. C++.</p>

c0d3r43v3r: <p>Hi there eqy<br>
Thanks for the inputs, i’ve noticed that when i lower the resolution of the input images all works fine and runs for 1 hours without any problems.<br>
The python pytorch implementation doesn’t exhibit the problem.<br>
It seems that libtorch is reserving memory in a different way than pytorch, it could be that the memory is fragmented or something else that is specific to the memory management utilization between libtorch and pytorch.</p>