Model weights memory corruption on cuda

ariel1: <p>Hi,<br>
After initializing a model and sending it to cuda device I see random changes of the weights of a simple model. It doesn’t happen when using CPU device.</p>
<p>Running on ubuntu 2020 LTS and 2022 LTS 64bit.<br>
NVIDIA GeForce RTX 2070<br>
Python 3.8 &amp; 3.10.6<br>
torch version ‘2.0.0+cu117’  (from pip)</p>
<p>Any ideas how to solve this issue?</p>
<p>Code to reproduce:</p>
<pre><code class="lang-auto">from typing import Tuple

import torch
from torch import nn


class UNetTiny(nn.Module):

    def __init__(self, in_channels: int, out_channels: int, num_block_filters: Tuple[int]):

        super().__init__()

        padding_mode = 'replicate'

        # Blocks of down portion
        down_modules = [
            nn.Conv2d(in_channels=in_channels, out_channels=num_block_filters[0], kernel_size=3, stride=2, padding=1, padding_mode=padding_mode)
        ]

        for i in range(1, len(num_block_filters)):
            down_modules.append(
                nn.Conv2d(in_channels=num_block_filters[i-1], out_channels=num_block_filters[i], kernel_size=3, stride=2, padding=1, padding_mode=padding_mode)
            )

        self._down_modules = nn.ModuleList(down_modules)

        up_deconv_modules = list()

        for i in range(len(num_block_filters) - 1, 0, -1):
            up_deconv_modules.append(
                torch.nn.ConvTranspose2d(in_channels=num_block_filters[i], out_channels=num_block_filters[i - 1], output_padding=1, padding=1, kernel_size=3, stride=2),
            )

        self._up_deconv_modules = nn.ModuleList(up_deconv_modules)

        self._out_deconv = torch.nn.ConvTranspose2d(in_channels=num_block_filters[0], out_channels=out_channels, output_padding=1, padding=1, kernel_size=3, stride=2)

    def forward(self, x) -&gt; torch.Tensor:

        for l in self._down_modules:
            x = l(x)
            x = torch.nn.functional.relu(x)

        for l_deconv in self._up_deconv_modules:
            x = l_deconv(x)
            x = torch.nn.functional.relu(x)

        res = self._out_deconv(x)
        return res


def _analyze_weights(unet_model):

    for name, tmp_module in unet_model.named_modules():
        if type(tmp_module) not in (torch.nn.modules.conv.Conv2d, torch.nn.modules.conv.ConvTranspose2d):
            continue

        if tmp_module.weight.detach().abs().max().item() &gt; 1.0:
            print(f'{name} weight::\t{tmp_module.weight.detach().abs().max().item()}')

        if tmp_module.bias.detach().abs().max().item() &gt; 1.0:
            print(f'{name} bias::\t{tmp_module.bias.detach().abs().max().item()}')


def _check_memory_corruption():

    dev = "cuda:0" if torch.cuda.is_available() else "cpu"
    # dev = 'cpu'
    device = torch.device(dev)

    print('Using device: ', device)

    N = 100
    num_block_filters = (64, 128, 256)

    unet = UNetTiny(
        in_channels=1,
        out_channels=3,
        num_block_filters=num_block_filters
    )

    unet.to(device=device)
    unet.train()

    for i in range(N):
        print(f'weights on iteration {i}')
        _analyze_weights(unet)


if __name__ == '__main__':

    _check_memory_corruption()
</code></pre>
<p>The output I’m getting:</p>
<p>Using device:  cuda:0<br>
weights on iteration 0<br>
weights on iteration 1<br>
weights on iteration 2<br>
weights on iteration 3<br>
weights on iteration 4<br>
weights on iteration 5<br>
weights on iteration 6<br>
weights on iteration 7<br>
weights on iteration 8<br>
weights on iteration 9<br>
weights on iteration 10<br>
weights on iteration 11<br>
weights on iteration 12<br>
weights on iteration 13<br>
weights on iteration 14<br>
weights on iteration 15<br>
weights on iteration 16<br>
weights on iteration 17<br>
_up_deconv_modules.0 weight::	254.71319580078125<br>
weights on iteration 18<br>
weights on iteration 19<br>
weights on iteration 20<br>
weights on iteration 21<br>
weights on iteration 22<br>
weights on iteration 23<br>
_down_modules.2 weight::	1815.224853515625<br>
weights on iteration 24<br>
weights on iteration 25<br>
weights on iteration 26<br>
weights on iteration 27<br>
_down_modules.2 weight::	1815.224853515625<br>
weights on iteration 28<br>
_down_modules.2 weight::	1166.347900390625<br>
weights on iteration 29<br>
weights on iteration 30<br>
weights on iteration 31<br>
weights on iteration 32<br>
weights on iteration 33<br>
weights on iteration 34<br>
weights on iteration 35<br>
weights on iteration 36<br>
weights on iteration 37<br>
weights on iteration 38<br>
weights on iteration 39<br>
weights on iteration 40<br>
weights on iteration 41<br>
weights on iteration 42<br>
weights on iteration 43<br>
_up_deconv_modules.0 weight::	249.4268798828125<br>
weights on iteration 44<br>
weights on iteration 45<br>
weights on iteration 46<br>
weights on iteration 47<br>
weights on iteration 48<br>
weights on iteration 49<br>
weights on iteration 50<br>
weights on iteration 51<br>
_up_deconv_modules.0 weight::	457.2268981933594<br>
weights on iteration 52<br>
weights on iteration 53<br>
weights on iteration 54<br>
weights on iteration 55<br>
_down_modules.2 weight::	427.8851318359375<br>
weights on iteration 56<br>
weights on iteration 57<br>
weights on iteration 58<br>
weights on iteration 59<br>
weights on iteration 60<br>
weights on iteration 61<br>
weights on iteration 62<br>
_down_modules.2 weight::	1423.19970703125<br>
weights on iteration 63<br>
weights on iteration 64<br>
weights on iteration 65<br>
weights on iteration 66<br>
weights on iteration 67<br>
weights on iteration 68<br>
weights on iteration 69<br>
_up_deconv_modules.0 weight::	254.71319580078125<br>
weights on iteration 70<br>
weights on iteration 71<br>
weights on iteration 72<br>
weights on iteration 73<br>
weights on iteration 74<br>
weights on iteration 75<br>
weights on iteration 76<br>
weights on iteration 77<br>
weights on iteration 78<br>
weights on iteration 79<br>
weights on iteration 80<br>
weights on iteration 81<br>
weights on iteration 82<br>
weights on iteration 83<br>
weights on iteration 84<br>
weights on iteration 85<br>
weights on iteration 86<br>
weights on iteration 87<br>
weights on iteration 88<br>
weights on iteration 89<br>
weights on iteration 90<br>
weights on iteration 91<br>
_down_modules.2 weight::	1815.224853515625<br>
weights on iteration 92<br>
weights on iteration 93<br>
weights on iteration 94<br>
weights on iteration 95<br>
weights on iteration 96<br>
weights on iteration 97<br>
weights on iteration 98<br>
weights on iteration 99</p>
<p>Process finished with exit code 0</p>

ptrblck: <p>Did you compare the parameters between both devices or are you assuming the max. values are guaranteed to never exceed your specified threshold? In any case, you might want to check if disabling IOMMU might help.</p>

ariel1: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="178845">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>disabling IOMMU</p>
</blockquote>
</aside>
<p>I did. The attached code is a simple program to show the issue. I also trained identical models on CPU and GPU simultaneously (loading the CPU model state to the GPU one) and the maximal difference among all the parameters was in the order of 10-7. After a couple of iterations suddenly it was ~1k<br>
even though the gradients of all the layers were almost identical.<br>
I will try the IOMMU option.<br>
One more question - is there a tool I can use inorder to catch events of GPU memory changes within the hardware level so I can inspect when and why it is changing?</p>