Change BN to GN in efficientnet

mobassir94: <p>like this post : <a href="https://discuss.pytorch.org/t/change-the-bn-to-gn-in-resnet/66088" class="inline-onebox">Change the BN to GN in resnet</a> i was trying groupnormalization for efficientnet ,<br>
my  model code is :</p>
<pre><code class="lang-auto">out_dim = 5
enet_type = 'efficientnet-b0'

pretrained_model = {
    'efficientnet-b0': '../input/efficientnet-pytorch/efficientnet-b0-08094119.pth'
}

    
class enetv2(nn.Module):
    def __init__(self, backbone, out_dim):
        super(enetv2, self).__init__()
        self.enet = enet.EfficientNet.from_name(backbone)
        self.enet.load_state_dict(torch.load(pretrained_model[backbone]))

        self.myfc = nn.Linear(self.enet._fc.in_features, out_dim)
        self.enet._fc = nn.Identity()

    def extract(self, x):
        return self.enet(x)

    def forward(self, x):
        x = self.extract(x)
        x = self.myfc(x)
        return x
model = enetv2(enet_type, out_dim=out_dim)
model = model.to(device)
</code></pre>
<p>if i try model.enet._bn0<br>
then it gives me this output :<br>
BatchNorm2d(32, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)</p>
<p>but with this code :</p>
<pre><code class="lang-auto">for name, module in model.named_modules():
    if isinstance(module, nn.BatchNorm2d):
        # Get current bn layer
        bn = getattr(model, name)
        # Create new gn layer
        gn = nn.GroupNorm(1, bn.num_features)
        # Assign gn
        print('Swapping {} with {}'.format(bn, gn))
        setattr(model, name, gn)

print(model)
</code></pre>
<p>i get this error :</p>
<pre><code class="lang-auto">AttributeError Traceback (most recent call last)
in
2 if isinstance(module, nn.BatchNorm2d):
3 # Get current bn layer
----&gt; 4 bn = getattr(model, name)
5 # Create new gn layer
6 gn = nn.GroupNorm(1, bn.num_features)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in getattr(self, name)
592 return modules[name]
593 raise AttributeError("’{}’ object has no attribute ‘{}’".format(
–&gt; 594 type(self).name, name))
595
596 def setattr(self, name, value):

AttributeError: ‘enetv2’ object has no attribute ‘enet._bn0’
</code></pre>
<p>now how do i replace BN layers with GN layers in my model?</p>

ptrblck: <p>Seems to be already <a href="https://discuss.pytorch.org/t/change-the-bn-to-gn-in-resnet/66088/11">solved</a>.</p>

mobassir94: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i am getting nan now with that, previously i reported that it’s working because i saw training began but today after 1st and 2nd epoch i can see loss getting nan and after second epoch giving me this error : ValueError: Input contains NaN, infinity or a value too large for dtype(‘float32’).</p>
<p>same kernel worked fine for batchnorm<br>
i just changed to groupnorm and started getting that error,here is the  code i used to change batchnorm with groupnorm :</p>
<pre><code class="lang-auto">for  i in range(0,len(model.enet._blocks)):
    try:
        print("...........found BN layer  so Replacing Bn0 with GN.........")
        gn0 = model.enet._blocks[i]._bn0.num_features
        model.enet._blocks[i]._bn0 = nn.GroupNorm(1,num_channels = gn0)
        print(model.enet._blocks[i]._bn0)
        
    except:
        print("BN layer Not  found!!")
        
    gn1 = model.enet._blocks[i]._bn1.num_features
    gn2 = model.enet._blocks[i]._bn2.num_features
    
    model.enet._blocks[i]._bn1 = nn.GroupNorm(1,num_channels = gn1)
    model.enet._blocks[i]._bn2 = nn.GroupNorm(1,num_channels = gn2)
    
    print(model.enet._blocks[i]._bn1)
    print(model.enet._blocks[i]._bn2)
    
</code></pre>

ptrblck: <p>Add <code>torch.autograd.detect_anomaly(True)</code> at the beginning of your script and post the stack trace here, please.</p>

mobassir94: <p>you mean to add that line of code at 1st cell of my notebook and train again?</p>

mobassir94: <h2>when i try torch.autograd.detect_anomaly(True)</h2>
<p>TypeError                                 Traceback (most recent call last)<br>
 in </p>
<p>TypeError: <strong>init</strong>() takes 1 positional argument but 2 were given</p>
<p>torch.autograd.set_detect_anomaly() works so i am trying it</p>

mobassir94: <p>full traceback :</p>
<pre><code class="lang-auto">
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;timed exec&gt; in &lt;module&gt;

&lt;ipython-input-17-b768044886f5&gt; in val_epoch(loader, get_output)
     49     acc = (PREDS == TARGETS).mean() * 100.
     50 
---&gt; 51     qwk = cohen_kappa_score(PREDS, TARGETS, weights='quadratic')
     52     qwk_k = cohen_kappa_score(PREDS[df_valid['data_provider'] == 'karolinska'], df_valid[df_valid['data_provider'] == 'karolinska'].isup_grade.values, weights='quadratic')
     53     qwk_r = cohen_kappa_score(PREDS[df_valid['data_provider'] == 'radboud'], df_valid[df_valid['data_provider'] == 'radboud'].isup_grade.values, weights='quadratic')

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py in cohen_kappa_score(y1, y2, labels, weights, sample_weight)
    583     """
    584     confusion = confusion_matrix(y1, y2, labels=labels,
--&gt; 585                                  sample_weight=sample_weight)
    586     n_classes = confusion.shape[0]
    587     sum0 = np.sum(confusion, axis=0)

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py in confusion_matrix(y_true, y_pred, labels, sample_weight, normalize)
    266 
    267     """
--&gt; 268     y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    269     if y_type not in ("binary", "multiclass"):
    270         raise ValueError("%s is not supported" % y_type)

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py in _check_targets(y_true, y_pred)
     79     """
     80     check_consistent_length(y_true, y_pred)
---&gt; 81     type_true = type_of_target(y_true)
     82     type_pred = type_of_target(y_pred)
     83 

/opt/conda/lib/python3.7/site-packages/sklearn/utils/multiclass.py in type_of_target(y)
    282     if y.dtype.kind == 'f' and np.any(y != y.astype(int)):
    283         # [.1, .2, 3] or [[.1, .2, 3]] or [[1., .2]] and not [1., 2., 3.]
--&gt; 284         _assert_all_finite(y)
    285         return 'continuous' + suffix
    286 

/opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
     58                     msg_err.format
     59                     (type_err,
---&gt; 60                      msg_dtype if msg_dtype is not None else X.dtype)
     61             )
     62     # for object dtype data, we only check for NaNs (GH-13254)

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
</code></pre>

mobassir94: <h1>train and val loop</h1>
<pre><code class="lang-auto">%%time
torch.autograd.detect_anomaly()
qwk_max = 0.
best_file = f'{kernel_type}_best_fold{fold}.pth'
for epoch in range(1, n_epochs+1):
    print(time.ctime(), 'Epoch:', epoch)
    scheduler.step(epoch-1)

    train_loss = train_epoch(train_loader, optimizer)
    val_loss, acc, qwk = val_epoch(valid_loader)

    content = time.ctime() + ' ' + f'Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, train loss: {np.mean(train_loss):.5f}, val loss: {np.mean(val_loss):.5f}, acc: {(acc):.5f}, qwk: {(qwk):.5f}'
    print(content)
    with open(f'log_{kernel_type}.txt', 'a') as appender:
        appender.write(content + '\n')

    if qwk &gt; qwk_max:
        print('score2 ({:.6f} --&gt; {:.6f}).  Saving model ...'.format(qwk_max, qwk))
        torch.save(model.state_dict(), best_file)
        qwk_max = qwk

torch.save(model.state_dict(), os.path.join(f'{kernel_type}_final_fold{fold}.pth'))
</code></pre>

ptrblck: <p>Yes, sorry <code>torch.autograd.set_detect_anomaly(True)</code> should be used.<br>
The previous error seems to be raised by <code>sklearn</code> and it seems the output of your model is already containing NaNs.</p>

mobassir94: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please can you tell me why my model outputting nan?<br>
if i use same model without groupnormalization code then i don’t get any  nan, only thing i changed in my code is bn to gn and nothing else! why my model is outputting nan after using groupnormalization?</p>

mobassir94: <p>also if you think this implementation is wrong,then  can you please give me correct code for changing bn to gn? :</p>
<pre><code class="lang-auto">for  i in range(0,len(model.enet._blocks)):
    try:
        print("...........found BN layer  so Replacing Bn0 with GN.........")
        gn0 = model.enet._blocks[i]._bn0.num_features
        model.enet._blocks[i]._bn0 = nn.GroupNorm(1,num_channels = gn0)
        print(model.enet._blocks[i]._bn0)
        
    except:
        print("BN layer Not  found!!")
        
    gn1 = model.enet._blocks[i]._bn1.num_features
    gn2 = model.enet._blocks[i]._bn2.num_features
    
    model.enet._blocks[i]._bn1 = nn.GroupNorm(1,num_channels = gn1)
    model.enet._blocks[i]._bn2 = nn.GroupNorm(1,num_channels = gn2)
    
    print(model.enet._blocks[i]._bn1)
    print(model.enet._blocks[i]._bn2)
</code></pre>

Keunhoi_An: <p>I’ve solved nan issue by fixing the issue that the lr is too high or forgetting to add small value of epsilon. So please check this out.</p>

mobassir94: <p>my lr is 3e-5 for adam,isn’t it natural? also why i should try epsilon where without epsilon batchnorm worked well,also during training i can see everything going good but during test i get nan</p>

SAI_VENKATESH_CHILUK: <p>Can you please send me the whole code of How did you convert BN to GN for efficientNet. Thanks in advance</p>