Why don't explicit free cpu resource in CachingHostAllocator

username: Shan19900305
cooked: <p>Hi, boss.<br>
For CPU block resources are not free.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L181">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L181" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L181" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L181</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="171" style="counter-reset: li-counter 170 ;">
          <li>    at::cuda::detail::getDeviceIndexWithPrimaryContext();</li>
          <li>if (primary_ctx_device_index.has_value()) {</li>
          <li>  device_guard.reset_device(</li>
          <li>      at::Device(at::DeviceType::CUDA, *primary_ctx_device_index));</li>
          <li>}</li>
          <li></li>
          <li>// Round up the allocation to the nearest power of two to improve reuse.</li>
          <li>void* ptr = nullptr;</li>
          <li>C10_CUDA_CHECK(cudaHostAlloc(</li>
          <li>    &amp;ptr, c10::llvm::PowerOf2Ceil(size), cudaHostAllocDefault));</li>
          <li class="selected">auto block = new Block();</li>
          <li>block-&gt;size_ = c10::llvm::PowerOf2Ceil(size);</li>
          <li>block-&gt;ptr_ = ptr;</li>
          <li>block-&gt;allocated_ = true;</li>
          <li></li>
          <li>{</li>
          <li>  std::lock_guard&lt;std::mutex&gt; g(blocks_mutex_);</li>
          <li>  blocks_.insert(block);</li>
          <li>  ptr_to_block_.insert({block-&gt;ptr_, block});</li>
          <li>}</li>
          <li>return {block-&gt;ptr_, reinterpret_cast&lt;void*&gt;(block)};</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L179">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L179" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L179" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L179</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="169" style="counter-reset: li-counter 168 ;">
          <li>at::OptionalDeviceGuard device_guard;</li>
          <li>auto primary_ctx_device_index =</li>
          <li>    at::cuda::detail::getDeviceIndexWithPrimaryContext();</li>
          <li>if (primary_ctx_device_index.has_value()) {</li>
          <li>  device_guard.reset_device(</li>
          <li>      at::Device(at::DeviceType::CUDA, *primary_ctx_device_index));</li>
          <li>}</li>
          <li></li>
          <li>// Round up the allocation to the nearest power of two to improve reuse.</li>
          <li>void* ptr = nullptr;</li>
          <li class="selected">C10_CUDA_CHECK(cudaHostAlloc(</li>
          <li>    &amp;ptr, c10::llvm::PowerOf2Ceil(size), cudaHostAllocDefault));</li>
          <li>auto block = new Block();</li>
          <li>block-&gt;size_ = c10::llvm::PowerOf2Ceil(size);</li>
          <li>block-&gt;ptr_ = ptr;</li>
          <li>block-&gt;allocated_ = true;</li>
          <li></li>
          <li>{</li>
          <li>  std::lock_guard&lt;std::mutex&gt; g(blocks_mutex_);</li>
          <li>  blocks_.insert(block);</li>
          <li>  ptr_to_block_.insert({block-&gt;ptr_, block});</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Through there is a API empty_cache can release resource, but this API is not using now.</p>
<p>So why don’t explicit free those resources, and how to ensure resources are not leaked.</p>

username: ptrblck
cooked: <p>A caching allocator tries to avoid allocating and freeing memory for each use for performance reasons. Resources can be freed by explicitly clearing the cache or at the teardown. Also the <code>CUDACachingAllocator</code> will free the cache in case it’s running OOM before retrying the allocation again.</p>

username: Shan19900305
cooked: <p>Yep, this is ok. There are also two questions about this.</p>
<ol>
<li>CachingHostAllocator don’t free cache when OOM, this will be directly failed;<br>
2)According empty_cache function, just free cudaEvent and pin memory from cudaHostAlloc directly without any device sync, this is OK?<br>
<a href="https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L283" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/release/1.13/aten/src/ATen/cuda/CachingHostAllocator.cpp#L283</a></li>
</ol>

username: ptrblck
cooked: <p>Why would you need a device sync to free page-locked memory on the host?</p>

username: Shan19900305
cooked: <p>In async mode, page-locked memory maybe freed before kernel using this memory.<br>
After read the codes, i found this is safe, because CachingHostAllocator_recordEvent and cudaFreeHost keep page-locked memory alive until kernel finished.<br>
Thanks a lot.</p>