RTX 3070: AMP doesn't seem to be working

username: Roman_Glushko
cooked: <p>I have recently built a DL workstation based on GeForce RTX 3070 card.</p>
<p>The problem I have so far is that I could not get my graphical card to work with AMP.</p>
<p>I’m using PyTorch Lightning to enable AMP in my project which in turn uses PyTorch native AMP support. It works for me in Kaggle kernels, but not on my workstation. It doesn’t matter whenever I configure half or full precision,  the memory consumptions are the same (the batch size did not changed during my checks):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/429c92f9260087ffdd0c8cc2fcf5b08db11555ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/429c92f9260087ffdd0c8cc2fcf5b08db11555ec" title="Screenshot 2021-04-19 at 16.04.43"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/429c92f9260087ffdd0c8cc2fcf5b08db11555ec_2_673x500.png" alt="Screenshot 2021-04-19 at 16.04.43" data-base62-sha1="9vgYeBHZIQQQofv71PYDqcYBRkE" width="673" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/429c92f9260087ffdd0c8cc2fcf5b08db11555ec_2_673x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/429c92f9260087ffdd0c8cc2fcf5b08db11555ec_2_1009x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/429c92f9260087ffdd0c8cc2fcf5b08db11555ec_2_1346x1000.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-19 at 16.04.43</span><span class="informations">1452×1078 42.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is my PyTorch version:</p>
<pre><code class="lang-auto">PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 
  - PyTorch Version: 1.8.1+cu111
</code></pre>
<p>I have installed it via</p>
<pre><code class="lang-auto">pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.htm
</code></pre>
<p>Here is another piece of information from the PyTorch env command:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.10 (x86_64)
GCC version: (Ubuntu 10.2.0-13ubuntu1) 10.2.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 3070
Nvidia driver version: 460.56
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] pytorch-lightning==1.2.3
[pip3] torch==1.8.1+cu111
[pip3] torchvision==0.9.1+cu111
[conda] Could not collect
</code></pre>
<p>Do you have any ideas what could be wrong?</p>
<p>Thank you in advanced!</p>

username: ptrblck
cooked: <p><code>nvidia-smi</code> will report the used device memory by all processes and will also show the allocated and reserved memory in PyTorch.<br>
The memory saving depends on the actual model as well as the used algorithms and you could check the detailed memory stats via <code>torch.cuda.memory_summary()</code>.</p>

username: Roman_Glushko
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, Piotr <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"> Glad you replayed!</p>
<p>Could not get <code>torch.cuda.memory_summary()</code> working for some reasons. I have executed the training from one SSH connection and then tried to run <code>python3 -c 'import torch; print(torch.cuda.memory_summary())'</code> from another SSH connection. However, I got all zeroes which is weird:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/4/842f9a4c49501bc21066f14238c1716d62d4feb1.png" data-download-href="https://discuss.pytorch.org/uploads/default/842f9a4c49501bc21066f14238c1716d62d4feb1" title="Screenshot 2021-04-21 at 11.49.26"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/842f9a4c49501bc21066f14238c1716d62d4feb1_2_416x500.png" alt="Screenshot 2021-04-21 at 11.49.26" data-base62-sha1="iRn0TCJdxEPW2qskSZZQIJ5LHIB" width="416" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/842f9a4c49501bc21066f14238c1716d62d4feb1_2_416x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/842f9a4c49501bc21066f14238c1716d62d4feb1_2_624x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/842f9a4c49501bc21066f14238c1716d62d4feb1_2_832x1000.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-21 at 11.49.26</span><span class="informations">1396×1674 171 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>At the same time, <code>nvtop</code> and <code>nvidia-smi</code> showed that GPU was utilised:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/663f7e39f7fb30c232c036bd25d21b8619e64a11.png" data-download-href="https://discuss.pytorch.org/uploads/default/663f7e39f7fb30c232c036bd25d21b8619e64a11" title="Screenshot 2021-04-21 at 11.57.01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/663f7e39f7fb30c232c036bd25d21b8619e64a11_2_690x292.png" alt="Screenshot 2021-04-21 at 11.57.01" data-base62-sha1="eAwJAmiqjbWluUberZKthudAeFX" width="690" height="292" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/663f7e39f7fb30c232c036bd25d21b8619e64a11_2_690x292.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/663f7e39f7fb30c232c036bd25d21b8619e64a11_2_1035x438.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/663f7e39f7fb30c232c036bd25d21b8619e64a11_2_1380x584.png 2x" data-dominant-color="F8FBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-21 at 11.57.01</span><span class="informations">3692×1566 251 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/839b3e2c5ce109b6d8f12a948c0df64b7232e4b0.png" data-download-href="https://discuss.pytorch.org/uploads/default/839b3e2c5ce109b6d8f12a948c0df64b7232e4b0" title="Screenshot 2021-04-21 at 11.57.23"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/839b3e2c5ce109b6d8f12a948c0df64b7232e4b0_2_634x500.png" alt="Screenshot 2021-04-21 at 11.57.23" data-base62-sha1="iMf9ByBDaRG0izRJFOVXwP78e4M" width="634" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/839b3e2c5ce109b6d8f12a948c0df64b7232e4b0_2_634x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/839b3e2c5ce109b6d8f12a948c0df64b7232e4b0_2_951x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/839b3e2c5ce109b6d8f12a948c0df64b7232e4b0_2_1268x1000.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-21 at 11.57.23</span><span class="informations">1440×1134 44.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In any case, I think that it doesn’t seem to be just a nvidia-smi information misleading.</p>
<p>Namely, I checked that my biggest batch size on the current dataset with FP32 is 8 samples. 9 gives OOM errors. When I enable FP16, the max batch_size still seems to be 8 and 9 still gives me OOM errors.</p>
<p>My model is PyTorch vanilla implementation of the RetinaNet (torchvision.models.detection.retinanet_resnet50_fpn + torchvision.models.detection.retinanet.RetinaNetHead) which is applied to a custom image dataset.</p>
<p>Is there another quick way to check that FP16 is really working on my env (to exclude a case when my particular usage of RetinaNet is somehow faulty)?</p>

username: cpeters
cooked: <p>Could it be that the data/gradients aren’t being cast to the half-precision variants? I don’t know how lightning works or if it’s supposed to include this automatically, but if autocast isn’t included then it won’t matter that AMP is imported.</p>

username: Roman_Glushko
cooked: <p>Hi <a class="mention" href="/u/cpeters">@cpeters</a> <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"><br>
When I did FP16 with PyTorch Lightning in Kaggle kernels, the precision config was the only change I did to enable AMP. However, indeed, the dataset was 8bits images.</p>
<p>This time I’m running on a dataset that I created myself (using just a mobile camera). So possibly this is the reason of the issue.</p>
<p>Does it mean that I need to include bits downscaling into my data processing routine?</p>
<p>…just in case, I have posted a thread on the PyTorch Lightning side about this issue:</p>
<ul>
<li><a href="https://github.com/PyTorchLightning/pytorch-lightning/discussions/7098" class="inline-onebox" rel="noopener nofollow ugc">AMP: Native AMP doesn't work in my Gigabyte Aorus Master RTX 3070 environment · Discussion #7098 · PyTorchLightning/pytorch-lightning · GitHub</a></li>
</ul>
<p>Thank you,<br>
Roman</p>

username: cpeters
cooked: <p>It sounds like it should be a problem on Lightning’s end? There’s nothing about the 3000 series that doesn’t work with AMP (I’ve done it here), and when writing native pytorch <a href="https://pytorch.org/docs/stable/amp.html#id4" rel="noopener nofollow ugc">the casting is done explicitly</a>.</p>

username: Roman_Glushko
cooked: <p><a class="mention" href="/u/cpeters">@cpeters</a>, Most likely. It’s getting interesting.</p>
<p>I have executed the project, which I used to test AMP in Kaggle kernels, in my workstation. AMP seems to be working there. So yeah, seems like it’s some issues on Lighting side which my combination of model/dataset discovered <img src="https://discuss.pytorch.org/images/emoji/apple/relieved.png?v=9" title=":relieved:" class="emoji" alt=":relieved:"></p>
<p>In any case, thank you for the help!</p>