Problems Using BatchNorm1d

username: nalkpas
cooked: <p>Hey, I’m trying to use BatchNorm1d in a simple neural network of a bunch of fully connected layers with a 1-dimensional input. My code is similar to the code in the tutorials, but it doesn’t run for some reason. I get the following error:</p>
<p>RuntimeError: running_mean should contain 1 elements not 32</p>
<p>Searching around for related terms hasn’t turned up much. When I try setting track_running_stats to false in the declaration, I get the error that track_running_stats isn’t a valid setting. I’m pretty confused and would appreciate some help.</p>
<p>Pictures of my code:<br>
            </p><div class="onebox imgur-album">
              <a href="https://imgur.com/a/pdlN8" target="_blank" rel="nofollow noopener">
                <span class="outer-box" style="width:600px">
                  <span class="inner-box">
                    <span class="album-title">[Album] Imgur</span>
                  </span>
                </span>
                <img src="https://i.imgur.com/j3nGqFY.jpg?fb" title="Imgur" height="315" width="600">
              </a>
            </div>

<p>Thanks!</p>

username: jpeg729
cooked: <p>You should be looking at the docs for the version of pytorch that you have installed, not the docs for the development version.<br>
If you have pytorch 0.3.1, then the docs are at <a href="http://pytorch.org/docs/0.3.1/">http://pytorch.org/docs/0.3.1/</a></p>
<p>What shape is the input data?</p>

username: nalkpas
cooked: <p>Oh, thanks for pointing that out. I didn’t know documentation defaulted to the development version.</p>
<p>A single example of my input data is just a vector of 17 numerical features. I’m passing in a minibatch of 100 examples, so the final input to the network is should be 100 x 16. The batchnorm layer takes 32 activations from the first layer, which I think is where the 32 in the error message is coming from.</p>

username: Juans
cooked: <p>Hi,<br>
I have the exact same problem. Did you find a solution? Could you please post how you solved the problem, if you did.<br>
Thanks!</p>

username: Ahmad_Momani
cooked: <p>what the size of the tensor going to batch normalization<br>
maybe you are trying to use batch normalization without giving it a batch<br>
when you do not have a batch freeze batch normalization using:<br>
state = torch.from_numpy(state).float()<br>
NetworkName.eval() # Sets the module in evaluation mode (no batchnorm or dropout)<br>
with torch.no_grad(): # no need to calculate gradients now, we are not training (save resources)<br>
action_values =  NetworkName(state)<br>
NetworkName.train() # # Sets the module back in training mode</p>
<p>when you have a batch use:<br>
NetworkName.forward(states)</p>

username: Bibonaut
cooked: <p>We had the same error message but it was related to the difference in the batch dimensions of the linear layer and the batch normalization layer. As far as I understand from documentation,</p>
<ul>
<li>nn.Linear uses batch dimensions such as [batch_size, *, features], where * can be any number of additional dimensions. (<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Linear" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Linear</a>)</li>
<li>nn.BatchNorm1d uses batch dimensions such as [batch_size, features, optional third dim] (<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm1d" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm1d</a>)</li>
</ul>
<p>If your batch has more than two dimensions and is processed by a linear layer, features are in the last dimension. To pass it to the batch normalization layer, you can flatten the tensor and reshape it after batch normalisation.<br>
Snippet:</p>
<pre><code class="lang-auto">x = linear(x)
x = batchnorm1d(x.flatten(0,-2)).reshape(x.shape)
</code></pre>
<p>I know you stated that your batch only has two dimensions. But I came across this thread while searching for a solution to my problem and perhaps my answer is helpful to someone.</p>