How to set per process memory fraction in c++?

chao_zhao: <p>When i load model and infer by libTorch, how could i set per process memory fraction in c++, like “torch.cuda.set_per_process_memory_fraction” in python.</p>

eqy: <p>I would check if you could call this caching allocator function that the Python binding calls: <a href="https://github.com/pytorch/pytorch/blob/f84f89b1c3f2bc74512e7a7b05ae6185164a9b3e/torch/csrc/cuda/Module.cpp#L509" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Module.cpp at f84f89b1c3f2bc74512e7a7b05ae6185164a9b3e · pytorch/pytorch · GitHub</a></p>

chao_zhao: <p><img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20">Tanks very much. It’s important for us.</p>

chao_zhao: <p>Is there any updates?</p>

ptrblck: <p>I believe <a class="mention" href="/u/eqy">@eqy</a> suggested to try to use <code>c10::cuda::CUDACachingAllocator::setMemoryFraction(fraction, device);</code> as shown in the link, so were you able to use it?</p>

chao_zhao: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="171692">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>setMemoryFraction</p>
</blockquote>
</aside>
<p>Sorry, I didn’t found this api in libtorch before. But i find now.<br>
I will try. Thanks.</p>

chao_zhao: <p>It seems doesn’t work normally. I had set mem fracation to 10%. But it can alloc gpu mem 20%.<br>
Test codes:</p>
<pre><code class="lang-auto">int test_set_mem_limit() {
  std::cout &lt;&lt; "test_set_mem_limit" &lt;&lt; std::endl;
  c10::cuda::CUDACachingAllocator::init(1);
  c10::cuda::CUDACachingAllocator::setMemoryFraction(0.1, 0);
  c10::cuda::CUDACachingAllocator::emptyCache();
  long long total_memory = 11721506816;
  auto tmp_tensor = torch::empty({(long long)(total_memory * 0.2)}, torch::kFloat32);
  std::cout &lt;&lt; "tmp_tensor sizes: " &lt;&lt; tmp_tensor.sizes() &lt;&lt; std::endl; // tmp_tensor sizes: [2344301363] 
  std::this_thread::sleep_for(std::chrono::seconds(10));
  std::cout &lt;&lt; "test_set_mem_limit end" &lt;&lt; std::endl;
  return 0;
}
</code></pre>
<p>Is my codes wrong using apis?</p>

eqy: <p>It looks like you aren’t specifying the device when calling <code>torch::empty</code>, which would allocate the tensor using host memory which wouldn’t be limited by <code>setMemoryFraction</code>. Could you try setting the device to CUDA for the call to <code>torch::empty</code>? Relevant documentation: <a href="https://pytorch.org/cppdocs/notes/tensor_creation.html#configuring-properties-of-the-tensor" class="inline-onebox" rel="noopener nofollow ugc">Tensor Creation API — PyTorch master documentation</a></p>

chao_zhao: <aside class="quote no-group" data-username="eqy" data-post="8" data-topic="171692">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>Tensor Creation API — PyTorch master documentation</p>
</blockquote>
</aside>
<p>It work. Thanks:</p>
<pre><code class="lang-auto">std::cout &lt;&lt; "test_set_mem_limit" &lt;&lt; std::endl;
  c10::cuda::CUDACachingAllocator::init(1);
  c10::cuda::CUDACachingAllocator::setMemoryFraction(0.1, 0);
  c10::cuda::CUDACachingAllocator::emptyCache();
  long long total_memory = 11721506816;
  auto tmp_tensor = torch::empty({(long long)(total_memory * 0.2)}, torch::kFloat32).to({torch::kCUDA, 0}); // CUDA out of memory. Tried to allocate 8.73 GiB (GPU 0; 10.92 GiB total capacity; 0 bytes already allocated; 5.97 GiB free; 1.09 GiB allowed;
  std::cout &lt;&lt; "tmp_tensor sizes: " &lt;&lt; tmp_tensor.sizes() &lt;&lt; std::endl;
  std::this_thread::sleep_for(std::chrono::seconds(10));
  std::cout &lt;&lt; "test_set_mem_limit end" &lt;&lt; std::endl;
</code></pre>