V100 compatibility

username: Hannibal046
cooked: <p>Hi, I build PyTorch from source by</p>
<pre><code class="lang-auto">TORCH_CUDA_ARCH_LIST="3.7 5.0 6.0 7.0 7.5 8.0 8.6"  python setup.py install
</code></pre>
<p>But why it can not run on V100 ? This pytorch version can run in A100 machine. And I notice that the <code>RuntimeError</code> happened in optimization step given by <code>Apex</code>, and this is how I install apex after pytorch</p>
<pre><code class="lang-auto">git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2ba28cb12f29fbb118d850461eb8a19b6713d47.png" data-download-href="https://discuss.pytorch.org/uploads/default/c2ba28cb12f29fbb118d850461eb8a19b6713d47" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2ba28cb12f29fbb118d850461eb8a19b6713d47_2_690x103.png" alt="image" data-base62-sha1="rMDtnptrLTvxyUHmG2jFqNfVwft" width="690" height="103" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2ba28cb12f29fbb118d850461eb8a19b6713d47_2_690x103.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2ba28cb12f29fbb118d850461eb8a19b6713d47_2_1035x154.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2ba28cb12f29fbb118d850461eb8a19b6713d47.png 2x" data-dominant-color="0C520C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1274×192 20.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea9f881542b73ccb678f24a477ff212ea42c62b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea9f881542b73ccb678f24a477ff212ea42c62b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea9f881542b73ccb678f24a477ff212ea42c62b4_2_690x376.png" alt="image" data-base62-sha1="xtzwq3mftEidr0h3DeALXF5wbgU" width="690" height="376" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea9f881542b73ccb678f24a477ff212ea42c62b4_2_690x376.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea9f881542b73ccb678f24a477ff212ea42c62b4_2_1035x564.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea9f881542b73ccb678f24a477ff212ea42c62b4.png 2x" data-dominant-color="F0D9F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1254×684 59.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca82053eb4beb566a43b0b79d030e84a79cd0b91.png" data-download-href="https://discuss.pytorch.org/uploads/default/ca82053eb4beb566a43b0b79d030e84a79cd0b91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca82053eb4beb566a43b0b79d030e84a79cd0b91_2_690x176.png" alt="image" data-base62-sha1="sTt11YaGedr21Xm8kNH1akAkxot" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca82053eb4beb566a43b0b79d030e84a79cd0b91_2_690x176.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca82053eb4beb566a43b0b79d030e84a79cd0b91_2_1035x264.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca82053eb4beb566a43b0b79d030e84a79cd0b91_2_1380x352.png 2x" data-dominant-color="ED76ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1828×468 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This seems to be relevant with Apex installation. I found this, and I am trying</p><aside class="onebox githubissue" data-onebox-src="https://github.com/NVIDIA/apex/issues/605#issuecomment-554453001">
  <header class="source">

      <a href="https://github.com/NVIDIA/apex/issues/605#issuecomment-554453001" target="_blank" rel="noopener nofollow ugc">github.com/NVIDIA/apex</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVIDIA/apex/issues/605#issuecomment-554453001" target="_blank" rel="noopener nofollow ugc">FusedLayerNorm leads to RuntimeError: CUDA error: no kernel image is available for execution on the device</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-11-15" data-time="13:56:22" data-timezone="UTC">01:56PM - 15 Nov 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/yangkky" target="_blank" rel="noopener nofollow ugc">
          <img alt="yangkky" src="https://avatars.githubusercontent.com/u/13158104?v=4" class="onebox-avatar-inline" width="20" height="20">
          yangkky
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">After a GPU tensor goes through FusedLayerNorm, the next time that memory is acc<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">essed I get a `RuntimeError: CUDA error: no kernel image is available for execution on the device`. 

To reproduce: 

```python
import torch
from apex.normalization import FusedLayerNorm

norm = FusedLayerNorm(16)
device = torch.device('cuda:' + str(0))
norm = norm.to(device)

x = torch.randn(3, 4, 16)
x = x.to(device)
attended = norm(x)
print(x)
```

Other operations on `attended` or `x` will also raise the error. However, if I move `x` to the CPU, I can then proceed to use it without any problems. 

I'm running this on an AWS p3.2xlarge instance based on the [AWS Deep Learning AMI (Ubuntu 18.04) Version 25.0](https://aws.amazon.com/releasenotes/aws-deep-learning-ami-ubuntu-18-04-version-25-0/?tag=releasenotes%23keywords%23aws-deep-learning-amis). We've updated pytorch to 1.3.0, and installed GPUtil, Apex, and gpustat using the following commands: 

```
source activate pytorch_p36

# Update to the latest PyTorch 1.3 (but not CUDA 10.0 instead of 10.1, because the AMI/env doesn't have it installed)
conda install pytorch==1.3.0 torchvision==0.4.1 cudatoolkit=10.0 -c pytorch -y

# Install GPUtil
pip install GPUtil

# Install NVIDIA Apex
git clone https://github.com/NVIDIA/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

# Install gpustat
pip install gpustat
```

Doing the same thing on an aws p2.xlarge instance with the same changes to the environment does not cause the error.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>I guess you haven’t specified the architectures while building <code>apex</code> so would need to rebuild it.</p>

username: Hannibal046
cooked: <p>Yes ! Exactly, after specifying the architectures when building <code>apex</code>, it works! Thank you</p>