Force libtorch to use CUDA context

Nacho_Valladares: <p>I’m trying to integrate libtoch to load a model into my application. My application does many CUDA stauff before I load the model with libtorch. Thus, the CUDA context has already been created.</p>
<p>For some reason, even the CUDA context has already been created and the calling thread has already a valid context, when I call</p>
<p>torch::jit::script::Module module = torch::jit::load(“test.pt”);<br>
module.to(at::kCUDA);</p>
<p>a new context is created by libtorch. The new context is not event push to the stack of context, is overwritting the current context. I know because if after the module.to(at::kCUDA) I call cuCtxPopCurrent the current context is null.</p>
<p>This causes a lot of problems because I cannot interact with current allocated memory I have in my context.</p>
<p>Is there anyway I can initialize somehow the libtorch CUDA context to use mine?</p>
<p>Thanks</p>

albanD: <p>Is there a recommended behavior here <a class="mention" href="/u/ngimel">@ngimel</a> ?</p>

Nacho_Valladares: <p>According to CUDA programming guide best practices:</p>
<blockquote>
<p>A CUDA context is analogous to a CPU process. All resources and actions performed within the driver API are encapsulated inside a CUDA context, and the system automatically cleans up these resources when the context is destroyed. Besides objects such as modules and texture or surface references, each context has its own distinct address space. As a result, CUdeviceptr values from different contexts reference different memory locations.</p>
</blockquote>
<p>And</p>
<blockquote>
<p>While multiple contexts (and their associated resources such as global memory allocations) can be allocated concurrently on a given GPU, only one of these contexts can execute work at any given moment on that GPU; contexts sharing the same GPU are time-sliced. Creating additional contexts incurs memory overhead for per-context data and time overhead for context switching. Furthermore, the need for context switching can reduce utilization when work from several contexts could otherwise execute concurrently (see also <a href="https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#concurrent-kernel-execution" rel="noopener nofollow ugc">Concurrent Kernel Execution</a>). Therefore, it is best to avoid multiple contexts per GPU within the same CUDA application.</p>
</blockquote>

albanD: <p>Hi,</p>
<p>It might be the jit that does not handle the context as it should. Could you please file a github issue and put the <code>jit</code> label on it? Thanks !</p>

Nacho_Valladares: <p>Done:</p>
<p>First time entering a libTorch git issue, please review if I did correctly. I will provide some code example in short.</p>
<p>Cheers</p>

albanD: <p>Thanks for taking the time to file the issue. Could you link it here in case someone else stumble upon this same issue and wants to track the status of the issue.</p>

Nacho_Valladares: <p>Opened issue</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/31565#issue-541662945" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/31565#issue-541662945" target="_blank" rel="nofollow noopener">Force libtorch to use CUDA context</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-12-23" data-time="09:23:57" data-timezone="UTC">09:23AM - 23 Dec 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/nachovall" target="_blank" rel="nofollow noopener">
          <img alt="nachovall" src="https://avatars0.githubusercontent.com/u/6033898?v=4" class="onebox-avatar-inline" width="20" height="20">
          nachovall
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">I’m trying to integrate C++ libtoch to load a model into my application. My application does many CUDA staff before I...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>