Model.eval() and model.param.required_grad=False

username: Alva-2020
cooked: <p>Hi, I encountered a strange problem: when I set model.eval() in evaluation stage and extractor bottleneck feature from audio. The extracted embedding are all [Nan], but when I set model. train(). The embeddings are normal numbers.  If I set model.train() and model.param.required_grad =False, are the inference results accuracy? Thanks very much for your help</p>

username: srishti-git1110
cooked: <p>Could you please post an executable snippet that’d reproduce this error?</p>
<p>Certain layers like batchnorm etc. work differently in train mode, so it’s best to use eval for inferencing.</p>

username: soulitzer
cooked: <p>It depends on what layers you have in your model. Typically:<br>
If you have any dropout layers in your model, setting model.eval() will disable the masking.<br>
If you have any batch norm layers, your model will use the running stats instead of statistics from the current input.</p>
<p>You most likely want to debug why eval mode is failing for your model, so I’d second srishti’s request for a self-contained code snippet to reproduce the issue.</p>

username: Alva-2020
cooked: <p>Tanks for your reply. I found the problem occurs in BacthNorm1d line. `        self.conv = Conv1d(<br>
in_channels=in_channels,<br>
out_channels=out_channels,<br>
kernel_size=kernel_size,<br>
dilation=dilation,<br>
)<br>
self.activation = activation()<br>
self.norm = BatchNorm1d(input_size=out_channels)</p>
<pre><code>def forward(self, x):
    pdb.set_trace()
    for name, param in self.norm.named_parameters():
        print("name-----&gt;",name)
        print("params----&gt;",param)
    pdb.set_trace()
    return self.norm(self.activation(self.conv(x)))
</code></pre>
<p>text`<br>
The outputs of  self.conv(x) and self.activation() are ok. The params ‘weights’ and ‘bias’ in self.norm is ok, but any tensor go through the self.norm(x) will become Nan. When I set track_running_stats=False, the self.norm(x)  go well.</p>

username: Alva-2020
cooked: <p>I have another questions, If I set the track_running_stats of torch.nn.BatchNorm1d() to False in the training phase model.train() and the test phase model.eval(), will it reduce the system performance?</p>

username: soulitzer
cooked: <p>During eval the accuracy of your model can suffer if for example during evaluation you are passing in input with batch size of 1 because the activations are no longer normalized. It may not matter if you are using large batch sizes during evaluation.</p>

username: Alva-2020
cooked: <aside class="quote no-group" data-username="Alva-2020" data-post="4" data-topic="175681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alva-2020/48/34062_2.png" class="avatar"> Alva-2020:</div>
<blockquote>
<p>I found the problem occurs in BacthNorm1d line. ` self.conv = Conv1d(<br>
in_channels=in_channels,<br>
out_channels=out_channels,<br>
kernel_size=kernel_size,<br>
dilation=dilation,<br>
)<br>
self.activation = activation()<br>
self.norm = BatchNorm1d(input_size=out_channels)</p>
<pre><code class="lang-auto">def forward(self, x):
    pdb.set_trace()
    for name, param in self.norm.named_parameters():
        print("name-----&gt;",name)
        print("params----&gt;",param)
    pdb.set_trace()
    return self.norm(self.activation(self.conv(x)))
</code></pre>
<p>text`<br>
The outputs of self.conv(x) and self.activation() are ok. The params ‘weights’ and ‘bias’ in self.norm is ok, but any tensor go through the self.norm(x) will become Nan. When I set track_running_stats=False, the self.norm(x) go well.</p>
</blockquote>
</aside>
<p>Thank you. I got it.</p>