BatchNorm1d reimplementation needed - Expected more than 1 value per channel when training, got input size torch.Size([1, 1])

Johnny_Cash: <p>Hi, I have this problem with the following Module (this is from <a href="https://github.com/woshidandan/TANet/blob/main/code/AVA/train_nni.py" rel="noopener nofollow ugc">this</a> implementation):</p>
<pre><code class="lang-auto">class TargetNet(nn.Module):
    def __init__(self):
        super(TargetNet, self).__init__()

        # L2
        self.fc1 = nn.Linear(365, 100)
        for i, m_name in enumerate(self._modules):
            if i &gt; 2:
                nn.init.kaiming_normal_(self._modules[m_name].weight.data)
        self.bn1 = nn.BatchNorm1d(100).to(device) #.cuda()
        self.relu1 = nn.PReLU()
        self.drop1 = nn.Dropout(1 - 0.5)

        self.relu7 = nn.PReLU()
        self.relu7.to(device) #.cuda()
        self.sig = nn.Sigmoid()

    def forward(self, x, paras):
        q = self.fc1(x)
        q = self.bn1(q)
        q = self.relu1(q)
        q = self.drop1(q)

        self.lin = nn.Sequential(TargetFC(paras['res_last_out_w'], paras['res_last_out_b']))
        q = self.lin(q)
        bn7 = nn.BatchNorm1d(q.shape[0])
        bn7.to(device) # .cuda()
        q = bn7(q)
        q = self.relu7(q)

        return q
</code></pre>
<p>When running the network on a single image I get:</p>
<p><code>ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 1])</code></p>
<p>Although I am putting the network into eval mode, the bn7 BatchNorm1d still stays in training mode, hence the issue. When I put the bn7 directly into eval mode the inference works but the results are totally off. Since the BatchNorm is initialized in the forward pass, I guess it uses the default parameters, since they are not trained. How could I reimplement this part to make this work on a single image forward pass?</p>
<p>The reason why I try to do this is on a single image without DataLoader is that I want to convert the model to ONNX, but there I need to pass through a dummy image, which in this case is not possible.</p>

ptrblck: <aside class="quote no-group" data-username="Johnny_Cash" data-post="1" data-topic="186488">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/johnny_cash/48/46016_2.png" class="avatar"> Johnny_Cash:</div>
<blockquote>
<p>Although I am putting the network into eval mode, the bn7 BatchNorm1d still stays in training mode, hence the issue.</p>
</blockquote>
</aside>
<p>This is expected since you are not creating <code>bn7</code> in the <code>__init__</code> method and are thus not properly registering this module. Each forward pass will create a new <code>bn7 = nn.BatchNorm1d(q.shape[0])</code> layer in <code>training</code> mode and will apply it the <code>q</code>. Note that no training will be performed on this layer since the next <code>forward</code> pass will re-create a new layer.</p>

Johnny_Cash: <p>Yes, exactly, but since the weights are provided and I can’t retrain the network I would like to use this (not clean) implementation  Hence I would like to implement BatchNorm1D in such a way that would work here - thus for batch size 1. I assume that the trainable parameters (which are obviously not trained here) are the default values when a BatchNorm1D is initialized,</p>

ptrblck: <p>You could call <code>bn7.train(mode=self.training)</code>, but note that a newly initialized bn layer in <code>eval</code> mode won’t do anything. It will just subtract zeros and divide by ones, so it’s a waste in compute.</p>

Johnny_Cash: <p>Thank you, I just tried that and the inference works with single image, but for some reason the result is still the same as when I put it into eval mode. This is how I tried:</p>
<pre><code class="lang-auto">bn7 = nn.BatchNorm1d(q.shape[0])
bn7.train(mode=self.training)
q = bn7(q)
</code></pre>

ptrblck: <p>Yes, the batchnorm layer won’t change its inputs since it wasn’t trained at all and since it’s in <code>eval</code> mode. Or are you referring to the error when describing “the result is still the same”?</p>

Johnny_Cash: <p>I am trying to say that the output of the model I get with your method is the same result as when I put it into eval mode (the whole model and the bn7 layer) instead.</p>
<p>As I establieshed in the beginning the ValueError occurs when my model is in eval mode and I dont do anything to the bn7 layer, which makes sense since it is in training mode.</p>
<p>It also doesn’t make sense put the bn7 layer into eval mode, as you also mentioned.</p>
<p>I would like to make the inference through bn7 with the values it has while training, because this is what the authors intend to do here I guess. Hence why I thought about implementing the functionality of the BatchNorm1D by myself, as how it would be in training mode, so I can run it when doing evaluation and not run into the initial valueerror.</p>
<p>The batchnorm1d seems to work just fine leaving it in training mode just when we use batch_size &gt;2, which makes sense. BatchNorm1D over batch_size of 1 would be pointless.</p>