Batch normalisation value overflow

Neural-Net-Ninja: <p><code>nn.BatchNorm1d()</code> throws error<br>
<code>lib\site-packages\torch\nn\functional.py", line 2450, in batch_norm return torch.batch_norm( RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</code><br>
when using  batch size of <code>8</code> or <code>16</code> or so, but worked fine when using <code>2</code> or <code>4</code> batch size.</p>
<p>now its strange, as the data is not so big, and I have 24gb vram.</p>
<p>what could be the cause ? and what can I do here ?</p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue, please?<br>
Also, could you post the output of <code>python -m torch.utils.collect_env</code>?</p>

Neural-Net-Ninja: <p>Hi, Thank you for your response.</p>
<p>code snippet</p>
<pre><code class="lang-auto">class BatchNorm(nn.Module):

    def __init__(self, input_dim: int, use_batch_normalization: bool = True, momentum: float = 0.1,
                 track_running_stats: bool = True) -&gt; None:

        super().__init__()
        self.input_dim = input_dim
        self.momentum = momentum
        self.use_batch_normalization = use_batch_normalization
        if self.use_batch_normalization:
            self.batch_norm = nn.BatchNorm1d(input_dim, momentum=momentum, track_running_stats=track_running_stats)
        else:
            self.bias = Parameter(torch.zeros(input_dim, dtype=torch.float32), requires_grad=True)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        """
        Forward pass of batch normalization block.

        :param x: Input of shape `(N, D)` or `(N, K, D)` where `N = number of points`, `K = number of neighbors`, and
            `D = number of feature channels`.
        :type x: torch.Tensor
        :return: Normalized output of the same shape as the input.
        :rtype: torch.Tensor
        """

        if self.use_batch_normalization:
            if x.dim() == 2:
                return self.batch_norm(x)
            if x.dim() == 3:
                # (N, K, D) -&gt; (N, D, K)
                x = x.transpose(1, 2).contiguous()
                # (N, D, K)
                output = self.batch_norm(x)
                # (N, D, K) -&gt; (N, K, D)
                return output.transpose(1, 2).contiguous()

            raise ValueError(f"Input dimension of batch normalization block should be 2 or 3, got {x.dim()}.")
        else:
            return x + self.bias

    def __repr__(self) -&gt; str:
        return 'BatchNormBlock(in_feat: {:d},' \
               ' momentum: {:.3f}, only_bias: {:s})'.format(self.input_dim,
                                                            self.momentum,
                                                            str(not self.use_batch_normalization))
</code></pre>
<p>which basically originate from</p>
<pre><code class="lang-auto">self.linear_pos_bias = nn.Sequential(OrderedDict([
                ('linear_1', nn.Linear(3, self.feature_dim, bias=False)),
                ('bn', BatchNorm(self.feature_dim)),
                ('relu', nn.ReLU(inplace=True)),
                ('linear_2', nn.Linear(self.feature_dim, self.feature_dim))
            ]))
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">  File "x:\Transformer.py", line 206, in forward
    peb = self.linear_pos_bias(pos)
  File "x:\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "x:\lib\site-packages\torch\nn\modules\container.py", line 204, in forward
    input = module(input)
  File "x:\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "x:\blocks\batch_norm.py", line 65, in forward
    output = self.batch_norm(x)
  File "x:\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "x:\lib\site-packages\torch\nn\modules\batchnorm.py", line 171, in forward
    return F.batch_norm(
  File "x:\lib\site-packages\torch\nn\functional.py", line 2450, in batch_norm
    return torch.batch_norm(
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
</code></pre>

Neural-Net-Ninja: <p>environment information:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.13.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.23.3
Libc version: N/A

Python version: 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.6.55
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 Ti
Nvidia driver version: 522.06
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.991
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.2
[pip3] torch==1.13.0+cu116
[pip3] torch-cluster==1.6.1+pt113cu116
[pip3] torch-geometric==2.3.1
[pip3] torch-scatter==2.1.1+pt113cu116
[pip3] torch-sparse==0.6.17+pt113cu116
[pip3] torch-spline-conv==1.2.2+pt113cu116
[pip3] torchaudio==0.13.0+cu116
[pip3] torchvision==0.14.0+cu116
[conda] Could not collect
</code></pre>

ptrblck: <p>Could you update to the latest PyTorch release and check if you would still see the issue?</p>

Neural-Net-Ninja: <p>Hello,</p>
<p>1.13.1+cu117 is the highest i can go making sure that all the tests pass in my code base, as the code base is yet to be migrated to the latest PyTorch version. which has to be done eventually, but till then if you could think of another workaround, please let me know!</p>
<p>thank you for your help.</p>
<pre><code class="lang-auto">PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.23.3
Libc version: N/A

Python version: 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 Ti
Nvidia driver version: 522.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==1.5.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.25.2
[pip3] torch==1.13.1+cu117
[pip3] torch-cluster==1.6.1+pt113cu117
[pip3] torch-geometric==2.3.1
[pip3] torch-scatter==2.1.1+pt113cu117
[pip3] torch-sparse==0.6.17+pt113cu117
[pip3] torch-spline-conv==1.2.2+pt113cu117
[pip3] torchaudio==0.13.1+cu117
[pip3] torchvision==0.14.1+cu117
[conda] Could not collect
</code></pre>