Eval function seems to not turn every layer to eval mode

Zenon_Lambrou: <p>Hello I am trying to train a CNN network. During training it trains fine but when I go to testing I get that error : “File “/.local/lib/python3.10/site-packages/torch/nn/functional.py”, line 2560, in group_norm<br>
_verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))<br>
File “/lib/python3.10/site-packages/torch/nn/functional.py”, line 2448, in _verify_batch_size<br>
raise ValueError(f"Expected more than 1 value per channel when training, got input size {size}”)" .</p>
<p>I understand is because the normalization layer expects a batch bigger than 1. I read online that using .eval() function should make the normalization layer accept a  batch of one. I am doing that but I still get the same error. Bellow is how I implement the layers of my model.</p>
<pre><code class="lang-auto">class ConvEncoder(nn.Module):
    def __init__(self, in_features, encoder_h=256, enc_width=(3, 2, 2, 2, 2, 2),
                 dropout=0., projection_head=False, enc_downsample=(3, 2, 2, 2, 2, 2),max_training_length=90):
        super().__init__()
        self.encoder = nn.Sequential()

        for i, (width, downsample) in enumerate(zip(enc_width, enc_downsample)):
            if i == 0:

                self.encoder.add_module("Encoder_{}".format(i), nn.Sequential(
                    nn.Conv1d(in_features, encoder_h, width, padding=width // 2),
                    nn.MaxPool1d(downsample),
                    nn.Dropout2d(dropout),
                    nn.GroupNorm(encoder_h , encoder_h),
                    nn.GELU(),
                ))
            else:

                self.encoder.add_module("Encoder_{}".format(i), nn.Sequential(
                    nn.Conv1d(in_features, encoder_h, width, padding=width // 2),
                    nn.MaxPool1d(downsample),
                    nn.Dropout2d(dropout),
                    nn.GroupNorm(encoder_h , encoder_h),
                    nn.GELU(),
                ))
            in_features = encoder_h

        if projection_head:
            self.encoder.add_module("projection-1", nn.Sequential(
                nn.Conv1d(in_features, in_features, 1),
                nn.Dropout2d(dropout*2),
                nn.GroupNorm(in_features, in_features),
                nn.GELU()
            ))
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Zenon_Lambrou" data-post="1" data-topic="203642">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zenon_lambrou/48/70377_2.png" class="avatar"> Zenon_Lambrou:</div>
<blockquote>
<p>I understand is because the normalization layer expects a batch bigger than 1. I read online that using .eval() function should make the normalization layer accept a batch of one.</p>
</blockquote>
</aside>
<p>I don’t think this is the case for <code>nn.GroupNorm</code> since according to the docs:</p>
<blockquote>
<p>This layer uses statistics computed from input data in both training and evaluation modes.</p>
</blockquote>
<p>which means that enough values are needed to compute the stats. If you cannot increase the batch size you could try to lower the <code>num_groups</code> values.<br>
Your code is unfortunately incomplete and not executable so I don’t know how the model is created and what the input is.</p>