Imagenet pytorch example Nan issues

rburgess: <p>Hi,</p>
<p>I am trying to adapt <a href="https://github.com/pytorch/examples/tree/main/imagenet/main.py" rel="noopener nofollow ugc">https://github.com/pytorch/examples/tree/main/imagenet/main.py</a> to Imagenet100. As my computer is not that spec’d up (2xRTX 2080) and the long training time, i decided to use Imagenet100 which meant i needed to make a slight change to resnet.py (<a href="https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py</a>). Please see <a href="https://github.com/rnangia/Imagenet100/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rnangia/Imagenet100: Altering Imagenet code from Pytorch examples to work with Imagenet100</a>.<br>
I had Nan issues. I thought it had something to do with an old version of Pytorch/CUDA/Cudnn; so I updated them. As it now stands, It is Cuda 11.8, Cudnn 8.9, torch 2.0.1+cu118.<br>
So, then I thought maybe it is a normalization issue. I checked <a href="https://github.com/danielchyeh/ImageNet-100-Pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - danielchyeh/ImageNet-100-Pytorch: (Pytorch) Training ResNets on ImageNet-100 data</a> and he uses the same imagenet normalization figures for 100 as it is for 1000 categories.<br>
I am lost where to go next from here. Please see attached logs.</p>
<pre><code class="lang-auto">Epoch: [2][401/986]	Time  0.391 ( 0.400)	Data  0.000 ( 0.038)	Loss 3.1367e+00 (3.4135e+00)	Acc@1  25.00 ( 17.94)	Acc@5  48.44 ( 44.55)
Epoch: [2][401/986]	Time  0.391 ( 0.396)	Data  0.000 ( 0.032)	Loss 3.1370e+00 (3.4206e+00)	Acc@1  25.00 ( 17.39)	Acc@5  43.75 ( 44.02)
Epoch: [2][411/986]	Time  0.405 ( 0.400)	Data  0.000 ( 0.037)	Loss 3.7681e+00 (3.4103e+00)	Acc@1  15.62 ( 18.01)	Acc@5  37.50 ( 44.63)
Epoch: [2][411/986]	Time  0.404 ( 0.396)	Data  0.000 ( 0.032)	Loss 3.2360e+00 (3.4179e+00)	Acc@1  23.44 ( 17.43)	Acc@5  50.00 ( 44.07)
Epoch: [2][421/986]	Time  0.404 ( 0.400)	Data  0.000 ( 0.037)	Loss 3.3991e+00 (3.4085e+00)	Acc@1  14.06 ( 17.99)	Acc@5  46.88 ( 44.65)
Epoch: [2][421/986]	Time  0.402 ( 0.396)	Data  0.000 ( 0.032)	Loss 3.2751e+00 (3.4179e+00)	Acc@1  20.31 ( 17.48)	Acc@5  46.88 ( 44.10)
Epoch: [2][431/986]	Time  0.389 ( 0.400)	Data  0.000 ( 0.037)	Loss 3.5405e+00 (3.4069e+00)	Acc@1  17.19 ( 18.02)	Acc@5  42.19 ( 44.69)
Epoch: [2][431/986]	Time  0.391 ( 0.396)	Data  0.000 ( 0.032)	Loss 3.2519e+00 (3.4150e+00)	Acc@1  28.12 ( 17.56)	Acc@5  46.88 ( 44.15)
Epoch: [2][441/986]	Time  0.383 ( 0.400)	Data  0.000 ( 0.037)	Loss 3.2058e+00 (3.4029e+00)	Acc@1  17.19 ( 18.09)	Acc@5  56.25 ( 44.83)
Epoch: [2][441/986]	Time  0.384 ( 0.396)	Data  0.000 ( 0.032)	Loss 3.5359e+00 (3.4134e+00)	Acc@1  18.75 ( 17.59)	Acc@5  39.06 ( 44.25)
Epoch: [2][451/986]	Time  0.372 ( 0.399)	Data  0.000 ( 0.037)	Loss nan (nan)	Acc@1   1.56 ( 17.87)	Acc@5   4.69 ( 44.34)
Epoch: [2][451/986]	Time  0.370 ( 0.396)	Data  0.000 ( 0.031)	Loss nan (nan)	Acc@1   0.00 ( 17.36)	Acc@5   1.56 ( 43.77)
Epoch: [2][461/986]	Time  0.367 ( 0.399)	Data  0.000 ( 0.036)	Loss nan (nan)	Acc@1   1.56 ( 17.50)	Acc@5   3.12 ( 43.51)
Epoch: [2][461/986]	Time  0.366 ( 0.395)	Data  0.000 ( 0.031)	Loss nan (nan)	Acc@1   6.25 ( 17.02)	Acc@5  10.94 ( 42.94)
Epoch: [2][471/986]	Time  0.367 ( 0.398)	Data  0.000 ( 0.036)	Loss nan (nan)	Acc@1   1.56 ( 17.14)	Acc@5   1.56 ( 42.69)
Epoch: [2][471/986]	Time  0.366 ( 0.394)	Data  0.000 ( 0.031)	Loss nan (nan)	Acc@1   0.00 ( 16.68)	Acc@5   3.12 ( 42.14)
Epoch: [2][481/986]	Time  0.367 ( 0.397)	Data  0.000 ( 0.036)	Loss nan (nan)	Acc@1   0.00 ( 16.81)	Acc@5   7.81 ( 41.89)
Epoch: [2][481/986]	Time  0.366 ( 0.394)	Data  0.000 ( 0.031)	Loss nan (nan)	Acc@1   3.12 ( 16.36)	Acc@5  15.62 ( 41.40)
Epoch: [2][491/986]	Time  0.368 ( 0.397)	Data  0.000 ( 0.036)	Loss nan (nan)	Acc@1   0.00 ( 16.48)	Acc@5   3.12 ( 41.12)
Epoch: [2][491/986]	Time  0.366 ( 0.393)	Data  0.000 ( 0.031)	Loss nan (nan)	Acc@1   1.56 ( 16.04)	Acc@5   4.69 ( 40.66)
Epoch: [2][501/986]	Time  0.367 ( 0.396)	Data  0.000 ( 0.035)	Loss nan (nan)	Acc@1   0.00 ( 16.16)	Acc@5   0.00 ( 40.38)
</code></pre>

ptrblck: <p>Check where the NaN value is created and e.g. if the forward pass of the model overflows or if it’s in the loss calculation etc. by printing values of intermediate tensors. Once you have narrowed down the operation causing the NaN (or Inf) value, you might get a better idea what is happening.<br>
Sometimes the input values already contain invalid values, so you might start with these. <code>torch.isfinite(tensor).all()</code> might be useful for your tests.</p>

rburgess: <p>I used the following block to test where the output is going to Nan</p>
<pre><code class="lang-auto">    def checking_iffinite(self, x, pp):
        if torch.isfinite(x).all():
            pass
        else:
            fl = open(os.path.join('outputs/jigpz_pretrain/run1', 'log.txt'), 'a')
            print("******Not finite*******", pp)
            fl.write("******Not finite*******"+ pp+'\n')
            fl.flush()
</code></pre>
<p>…</p>
<pre><code class="lang-auto">def _forward_impl(self, x: Tensor) -&gt; Tensor:
        # See note [TorchScript super()]
        x = self.conv1(x)
        self.checking_iffinite(x, "_forward_impl conv1")
        x = self.bn1(x)
        self.checking_iffinite(x, "_forward_impl bn1")
</code></pre>
<p>I found where the output is going to Nan</p>
<pre><code class="lang-auto">******Not finite*******Bottleneck identity
******Not finite*******Bottleneck identity
******Not finite*******Bottleneck conv1
******Not finite*******Bottleneck conv1
******Not finite*******Bottleneck bn1
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck bn1
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck conv2
******Not finite*******Bottleneck bn2
******Not finite*******Bottleneck conv2
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck bn2
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck conv3
******Not finite*******Bottleneck conv3
******Not finite*******Bottleneck bn3
******Not finite*******Bottleneck out
******Not finite*******Bottleneck bn3
******Not finite*******Bottleneck out relu
******Not finite*******Bottleneck out
******Not finite*******_forward_impl layer2
</code></pre>
<p>It is going to Nan in forward implementation at layer2. What should I do next?</p>

ptrblck: <p>Did you check the actual input as well? Try to narrow it down to an operation using valid inputs and creating invalid ones.</p>

rburgess: <p>Hello again,<br>
I had a slight typo (in the error tracing code). After correcting it, I found the place where the error is occuring. It seems like the batch normalization i.e. nn.BatchNorm2d in the forward of Bottleneck block of Resnet is the culprit. Whereto from here?</p>
<pre><code class="lang-auto">Epoch: [1][301/986]	Time  0.532 ( 0.539)	Data  0.000 ( 0.033)	Loss 3.7381e+00 (3.9519e+00)	Acc@1   9.38 (  9.18)	Acc@5  39.06 ( 28.29)
Epoch: [1][311/986]	Time  0.538 ( 0.544)	Data  0.000 ( 0.039)	Loss 3.7849e+00 (3.9466e+00)	Acc@1  14.06 (  9.09)	Acc@5  25.00 ( 29.02)
Epoch: [1][311/986]	Time  0.536 ( 0.539)	Data  0.000 ( 0.032)	Loss 3.8315e+00 (3.9480e+00)	Acc@1   9.38 (  9.20)	Acc@5  31.25 ( 28.37)
******Not finite*******Bottleneck bn1
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck bn2
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck conv3
******Not finite*******Bottleneck bn3
******Not finite*******Bottleneck out
******Not finite*******Bottleneck out relu
******Not finite*******Bottleneck identity
******Not finite*******Bottleneck conv1
******Not finite*******Bottleneck bn1
******Not finite*******Bottleneck relu
******Not finite*******Bottleneck bn2
</code></pre>
<p>…</p>
<pre><code class="lang-auto">******Not finite*******_forward_impl layer3
******Not finite*******_forward_impl layer4
******Not finite*******_forward_impl avgpool
******Not finite*******_forward_impl flatten
******Not finite*******_forward_impl fc
Output in Train is not finite ::: 315iteration
</code></pre>