PyTorch 1.2.0 with compute capability 3.5

username: tholzmann
cooked: <p>Hi,</p>
<p>I am using PyTorch 1.2.0 self-compiled with CUDA compute capability 5.2 with C++ and everything works as expected.</p>
<p>I read somewhere that everything down to compute capability 3.5 is supported. Hence, as we aim to support as many graphic cards as possible, i tried to compile PyTorch with compute capability 3.5. However, I get an error when using this version:</p>
<blockquote>
<p>.THCudaCheck FAIL file=D:/tools/pytorch-v1.2.0/aten/src\THC/generic/THCTensorMath.cu line=16 error=209 : no kernel image is available for execution on the device<br>
exception message: cuda runtime error (209) : no kernel image is available for execution on the device at D:/tools/pytorch-v1.2.0/aten/src\THC/generic/THCTensorMath.cu:16<br>
The above operation failed in interpreter, with the following stack trace:<br>
at code/model-input_rgbip-output_14classes_best_train_2019_08_03_cpu-eval-mode-export_latest_pytorch.py:292:12<br>
_135 = getattr(_131, “1”)<br>
_136 = _135.weight<br>
_137 = _135.bias<br>
_138 = getattr(self.decoder0, “0”)<br>
_139 = _138.weight<br>
_140 = _138.bias<br>
_141 = getattr(self.logit, “0”)<br>
_142 = _141.weight<br>
_143 = _141.bias<br>
input0 = torch._convolution(input, <em>1, None, [2, 2], [3, 3], [1, 1], False, [0, 0], 1, True, False, True)<br>
~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
input1 = torch.batch_norm(input0, weight, bias, running_mean, running_var, False, 0.10000000000000001, 1.0000000000000001e-05, True)<br>
input2 = torch.relu</em>(input1)<br>
input3 = torch.max_pool2d(input2, [3, 3], [2, 2], [1, 1], [1, 1], False)<br>
input4 = torch._convolution(input3, <em>5, None, [1, 1], [1, 1], [1, 1], False, [0, 0], 1, True, False, True)<br>
input5 = torch.batch_norm(input4, weight0, bias0, running_mean0, running_var0, False, 0.10000000000000001, 1.0000000000000001e-05, True)<br>
input6 = torch.relu</em>(input5)<br>
input7 = torch._convolution(input6, <em>7, None, [1, 1], [1, 1], [1, 1], False, [0, 0], 1, True, False, True)<br>
out = torch.batch_norm(input7, weight1, bias1, running_mean1, running_var1, False, 0.10000000000000001, 1.0000000000000001e-05, True)<br>
input8 = torch.add</em>(out, input3, alpha=1)Compiled from code /opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py(340): forward<br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py(523): _slow_forward<br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py(537): <strong>call</strong><br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py(92): forward<br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py(523): _slow_forward<br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py(537): <strong>call</strong><br>
…/dl/models/unet.py(153): forward<br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py(523): _slow_forward<br>
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py(537): <strong>call</strong><br>
/opt/conda/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py(883): trace_module<br>
/opt/conda/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py(751): trace<br>
(5): <br>
/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3296): run_code<br>
/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3214): run_ast_nodes<br>
/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3049): run_cell_async<br>
/opt/conda/lib/python3.6/site-packages/IPython/core/async_helpers.py(67): _pseudo_sync_runner<br>
/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2874): _run_cell<br>
/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2848): run_cell<br>
/opt/conda/lib/python3.6/site-packages/ipykernel/zmqshell.py(536): run_cell<br>
/opt/conda/lib/python3.6/site-packages/ipykernel/ipkernel.py(294): do_execute<br>
/opt/conda/lib/python3.6/site-packages/tornado/gen.py(209): wrapper<br>
/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py(534): execute_request<br>
/opt/conda/lib/python3.6/site-packages/tornado/gen.py(209): wrapper<br>
/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py(267): dispatch_shell<br>
/opt/conda/lib/python3.6/site-packages/tornado/gen.py(209): wrapper<br>
/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py(357): process_one<br>
/opt/conda/lib/python3.6/site-packages/tornado/gen.py(742): run<br>
/opt/conda/lib/python3.6/site-packages/tornado/gen.py(781): inner<br>
/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py(743): _run_callback<br>
/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py(690): <br>
/opt/conda/lib/python3.6/asyncio/events.py(145): _run<br>
/opt/conda/lib/python3.6/asyncio/base_events.py(1451): _run_once<br>
/opt/conda/lib/python3.6/asyncio/base_events.py(438): run_forever<br>
/opt/conda/lib/python3.6/site-packages/tornado/platform/asyncio.py(148): start<br>
/opt/conda/lib/python3.6/site-packages/ipykernel/kernelapp.py(505): start<br>
/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py(658): launch_instance<br>
/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py(16): <br>
/opt/conda/lib/python3.6/runpy.py(85): _run_code<br>
/opt/conda/lib/python3.6/runpy.py(193): _run_module_as_main</p>
</blockquote>
<p>So I assume compute capability 3.5 is also not fully supported anymore?<br>
Down to which compute capability PyTorch 1.2.0 should work correctly?</p>
<p>My system:<br>
Windows 10<br>
Visual Studio 2019 - CUDA 10.1<br>
Python 3.7<br>
Self-Compiled PyTorch 1.2.0</p>
<p>Thanks!</p>
<p>Best,<br>
Thomas</p>

username: tholzmann
cooked: <p>I tried to build with compute capability 5.0 now, and this build is working. So I assume PyTorch requires 5.0 as minimum compute capability. Is this correct?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The comments about pytorch working all the way down to cc3.5 are quite old. I’m afraid this is not true anymore.</p>

username: peterjc123
cooked: <p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/builder/blob/master/conda/pytorch-1.1.0/bld.bat#L18" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/builder/blob/master/conda/pytorch-1.1.0/bld.bat#L18" target="_blank" rel="nofollow noopener">pytorch/builder/blob/master/conda/pytorch-1.1.0/bld.bat#L18</a></h4>
<pre class="onebox"><code class="lang-bat"><ol class="start lines" start="8" style="counter-reset: li-counter 7 ;">
<li>    set build_with_cuda=</li>
<li>) else (</li>
<li>    set build_with_cuda=1</li>
<li>    set desired_cuda=%CUDA_VERSION:~0,-1%.%CUDA_VERSION:~-1,1%</li>
<li>)</li>
<li>
</li>
<li>if "%build_with_cuda%" == "" goto cuda_flags_end</li>
<li>
</li>
<li>set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%desired_cuda%</li>
<li>set CUDA_BIN_PATH=%CUDA_PATH%\bin</li>
<li class="selected">set TORCH_CUDA_ARCH_LIST=3.5;5.0+PTX</li>
<li>if "%desired_cuda%" == "8.0" set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1</li>
<li>if "%desired_cuda%" == "9.0" set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;7.0</li>
<li>if "%desired_cuda%" == "9.2" set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0</li>
<li>if "%desired_cuda%" == "10.0" set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0;7.5</li>
<li>set TORCH_NVCC_FLAGS=-Xfatbin -compress-all</li>
<li>
</li>
<li>:cuda_flags_end</li>
<li>
</li>
<li>set DISTUTILS_USE_SDK=1</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/builder/blob/master/windows/cuda100.bat#L36" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/builder/blob/master/windows/cuda100.bat#L36" target="_blank" rel="nofollow noopener">pytorch/builder/blob/master/windows/cuda100.bat#L36</a></h4>
<pre class="onebox"><code class="lang-bat"><ol class="start lines" start="26" style="counter-reset: li-counter 25 ;">
<li>    echo NVTX ^(Visual Studio Extension ^for CUDA^) ^not installed, failing</li>
<li>    exit /b 1</li>
<li>    goto optcheck</li>
<li>)</li>
<li>
</li>
<li>IF "%CUDA_PATH_V10_0%"=="" (</li>
<li>    echo CUDA 10.0 not found, failing</li>
<li>    exit /b 1</li>
<li>) ELSE (</li>
<li>    IF "%BUILD_VISION%" == "" (</li>
<li class="selected">        set TORCH_CUDA_ARCH_LIST=3.5;5.0+PTX;6.0;6.1;7.0;7.5</li>
<li>        set TORCH_NVCC_FLAGS=-Xfatbin -compress-all</li>
<li>    ) ELSE (</li>
<li>        set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_50,code=compute_50</li>
<li>    )</li>
<li>
</li>
<li>    set "CUDA_PATH=%CUDA_PATH_V10_0%"</li>
<li>    set "PATH=%CUDA_PATH_V10_0%\bin;%PATH%"</li>
<li>)</li>
<li>
</li>
<li>:optcheck</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/builder/blob/master/windows/cuda90.bat#L36" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/builder/blob/master/windows/cuda90.bat#L36" target="_blank" rel="nofollow noopener">pytorch/builder/blob/master/windows/cuda90.bat#L36</a></h4>
<pre class="onebox"><code class="lang-bat"><ol class="start lines" start="26" style="counter-reset: li-counter 25 ;">
<li>    echo NVTX ^(Visual Studio Extension ^for CUDA^) ^not installed, failing</li>
<li>    exit /b 1</li>
<li>    goto optcheck</li>
<li>)</li>
<li>
</li>
<li>IF "%CUDA_PATH_V9_0%"=="" (</li>
<li>    echo CUDA 9 not found, failing</li>
<li>    exit /b 1</li>
<li>) ELSE (</li>
<li>    IF "%BUILD_VISION%" == "" (</li>
<li class="selected">        set TORCH_CUDA_ARCH_LIST=3.5;5.0+PTX;6.0;7.0</li>
<li>        set TORCH_NVCC_FLAGS=-Xfatbin -compress-all</li>
<li>    ) ELSE (</li>
<li>        set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_50,code=compute_50</li>
<li>    )</li>
<li>
</li>
<li>    set "CUDA_PATH=%CUDA_PATH_V9_0%"</li>
<li>    set "PATH=%CUDA_PATH_V9_0%\bin;%PATH%"</li>
<li>)</li>
<li>
</li>
<li>:optcheck</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Actually 3.5 is in the CUDA_ARCH_LIST, I wonder why that is not supported.</p>

username: tholzmann
cooked: <p>Interestingly, when I use the prebuild windows version built with CUDA/CuDNN, it works on a graphics card with cc 3.7 (Tesla K80). However, when I build pytorch myself with CUDA but without CuDNN, it does not work on this graphics card.</p>
<p>Is it possible that some instructions are only implemented for a higher cc in CUDA, but if CuDNN is used these instructions are implemented with CuDNN and hence it works?</p>