torch::jit::script::Module::Forward failes (LoadLibraryA) on c++ static lib for windows

username: tkr
cooked: <p>Hi,</p>
<p>I am trying to train on linux (python) and do inference on windows with c++ static lib application.<br>
When calling torch::jit::script::Module::Forward(), following error occurs.<br>
The application with dll does not fail.</p>
<pre><code class="lang-auto">The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/Model.py", line 37, in forward
    _19 = (_6).forward((_7).forward((_8).forward(_18, ), ), )
    input0 = torch.cat([(_5).forward(_19, ), _15], 1)
    _20 = (_3).forward((_4).forward(input0, ), )
                        ~~~~~~~~~~~ &lt;--- HERE
    _21 = (_2).forward((_14).forward2(_20, ), )
    return (_0).forward((_1).forward(_21, ), )
  File "code/__torch__/CompactModel.py", line 36, in forward
    _18 = (_9).forward((_10).forward(_17, ), )
    _19 = (_6).forward((_7).forward((_8).forward(_18, ), ), )
    input0 = torch.cat([(_5).forward(_19, ), _15], 1)
             ~~~~~~~~~ &lt;--- HERE
    _20 = (_3).forward((_4).forward(input0, ), )
    _21 = (_2).forward((_14).forward2(_20, ), )

Traceback of TorchScript, original code (most recent call last):
/docker_share/source/Model.py(193): forward
/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py(534): _slow_forward
/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py(548): __call__
/usr/local/lib/python3.8/site-packages/torch/jit/__init__.py(1027): trace_module
/usr/local/lib/python3.8/site-packages/torch/jit/__init__.py(873): trace
./source/jitrace.py(700): &lt;module&gt;
RuntimeError: error in LoadLibraryA
</code></pre>
<p>The error message changed from “LoadLibraryA” to “GetProcAddress” when I placed “caffe2_nvrtc.dll” next to the application.  caffe2_nvrtc.dll is created under build/bin and caffe2_nvrtc.lib is not created.<br>
Is caffe2_nvrtc.dll related this problem ?</p>
<p>Do you have any suggestion ?<br>
Thank you.</p>

username: peterjc123
cooked: <p>Please copy all the DLLs to the directory of your application.</p>

username: tkr
cooked: <p>There are NO DLLs in my application dir.<br>
The cuda and cudnn libraries are in the $PATH.</p>
<p>Although I do not want to use dynamic link library,<br>
application succeeded by below.</p>
<ul>
<li>link official distributed “caffe2_nvrtc.lib”</li>
<li>place “caffe2_nvcrt.dll” next to the application.</li>
</ul>
<p>So, to succeed with static library, I tried below and it didn’t work.</p>
<ul>
<li>change CMakeFiles.txt under “caffe2” directory as below and run build.</li>
</ul>
<pre><code class="lang-auto">565c565
&lt;     add_library(caffe2_nvrtc SHARED ${ATen_NVRTC_STUB_SRCS})
---
&gt;     add_library(caffe2_nvrtc ${ATen_NVRTC_STUB_SRCS})
</code></pre>
<ul>
<li>caffe2_nvrtc.lib is created (and caffe2_nvrtc.dll is not).</li>
<li>link it to the app.</li>
</ul>
<p>But it does work if “caffe2_nvrtc.dll” (the official one) is placed next to the application.</p>
<p>The difference between the loaded DLLs when the application succeeds or fails are :</p>
<ul>
<li>caffe2_nvrtc.dll</li>
<li>nvrtc64_101_0.dll</li>
</ul>
<p>So what should I do to succeed with static cuffe2_nvrtc.lib?</p>

username: peterjc123
cooked: <p>Hey, we don’t provide static libs currently. You’ll have to build that from source.</p>

username: peterjc123
cooked: <aside class="quote no-group" data-username="tkr" data-post="3" data-topic="85977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tkr/48/16838_2.png" class="avatar"> tkr:</div>
<blockquote>
<p>cuffe2_nvrtc.lib</p>
</blockquote>
</aside>
<p>Unlike the relations between <code>.so</code> and <code>.a</code> on Linux, <code>.lib</code> files don’t necessarily refer to the file name of a static lib. It can also be an import library for the DLL.</p>

username: tkr
cooked: <blockquote>
<p>Hey, we don’t provide static libs currently. You’ll have to build that from source.</p>
</blockquote>
<p>Let me rephrase what I wanted to say…<br>
What I want to do is to create successful static version of libtorch.<br>
I have built the static  libtorch from source with “set BUILD_SHARED_LIBS=OFF” BUT</p>
<ul>
<li>cuffe2_nvrtc.lib is NOT created.</li>
<li>caffe2_nvrtc.dll is created under torch/bin.</li>
</ul>
<p>My assumption is below.<br>
“SHARED” is set in add_library and because of that, DLL is created.  At the same time,<br>
BUILD_SHARED_LIBS=OFF and dllexport is not defined and lib is not created. Files under lib are linked to the application and error occurs.</p>
<p>I now know by using official caffe2_nvrtc.lib and caffe2_nvrtc.dll, the application succeeds.<br>
I’d like to know how to create static libtorch library.</p>
<p>I’ve done rewriting CMakeList.txt and built static version of libtorch and</p>
<ul>
<li>caffe2_nvrtc.lib is created (and this should be static lib, right?)</li>
<li>caffe2_nvrtc.dll is NOT created.</li>
</ul>
<p>By linking above to the app, it will end up with the error I wrote in the beginning.<br>
By placing official caffe2_nvrtc.dll next to the app, it works. (meaning static lib is not created correctly ?)</p>
<p>Any suggestion to succeed my attempt?</p>

username: peterjc123
cooked: <p>Maybe some of the libs are optimized away. You could try passing <code>/WHOLEARCHIVE:caffe2_nvrtc.lib</code> in your project to force the linker to stop doing that.</p>

username: tkr
cooked: <p>I thought the same way.</p>
<ul>
<li>/WHOLEARCHIVE:caffe2_nvrtc.lib</li>
<li>link C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64\nvrtc.lib</li>
</ul>
<p>I tried above and still get the same error.<br>
The application does not load “nvrtc64_101_0.dll”</p>
<p>Any other suggestion?</p>

username: Scott_Hoang
cooked: <p>use WSL2 to run ubuntu on WINDOW!</p>

username: peterjc123
cooked: <p>You may need to link other CUDA libs as well, like cudart64_xxx.lib. But some of them doesn’t have static libs so you still need some DLLs.</p>

username: peterjc123
cooked: <p>Well, I don’t think WSL2 is a good choice for deployment.</p>

username: tkr
cooked: <p>I’ve tried few other things and below change on caffe2/CMakeList.txt worked!</p>
<pre><code class="lang-diff">diff --git a/caffe2/CMakeLists.txt b/caffe2/CMakeLists.txt
index 8025a7de3c..8e94978e72 100644
--- a/caffe2/CMakeLists.txt
+++ b/caffe2/CMakeLists.txt
@@ -561,7 +561,7 @@ if (NOT INTERN_BUILD_MOBILE OR NOT BUILD_CAFFE2_MOBILE)
       ${TORCH_SRC_DIR}/csrc/cuda/comm.cpp
       ${TORCH_SRC_DIR}/csrc/jit/tensorexpr/cuda_codegen.cpp
     )
-    add_library(caffe2_nvrtc SHARED ${ATen_NVRTC_STUB_SRCS})
+    add_library(caffe2_nvrtc ${ATen_NVRTC_STUB_SRCS})
     target_link_libraries(caffe2_nvrtc ${CUDA_NVRTC} ${CUDA_CUDA_LIB} ${CUDA_NVRTC_LIB})
     target_include_directories(caffe2_nvrtc PRIVATE ${CUDA_INCLUDE_DIRS})
     install(TARGETS caffe2_nvrtc DESTINATION "${TORCH_INSTALL_LIB_DIR}")
@@ -703,6 +703,9 @@ ELSEIF(USE_CUDA)
   cuda_add_library(torch_cuda ${Caffe2_GPU_SRCS})
   set(CUDA_LINK_LIBRARIES_KEYWORD)
   torch_compile_options(torch_cuda)  # see cmake/public/utils.cmake
+  if (NOT BUILD_SHARED_LIBS)
+    target_compile_definitions(torch_cuda PRIVATE USE_DIRECT_NVRTC)
+  endif()
   if (USE_NCCL)
     target_link_libraries(torch_cuda PRIVATE __caffe2_nccl)
</code></pre>
<p>In aten/src/ATen/cuda/detail/CUDAHooks.cpp, “<span class="hashtag">#ifdef</span> USE_DIRECT_NVRTC” directive is used.<br>
But “USE_DIRECT_NVRTC” was not defined in any CMakeList.txt and because of that,<br>
application linked with satic libtorch tries to load “caffe2_nvrtc.dll”.</p>

username: peterjc123
cooked: <aside class="quote no-group" data-username="tkr" data-post="12" data-topic="85977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tkr/48/16838_2.png" class="avatar"> tkr:</div>
<blockquote>
<p>USE_DIRECT_NVRTC</p>
</blockquote>
</aside>
<p>Interesting facts. Thanks for the finding.</p>