Why i got negative values for the feartures?

samm: <p>I got bad values for the validation loss as the first epoch has</p>
<blockquote>
<p>ep001 : loss: 4.473 val_loss : 5.635</p>
</blockquote>
<p>val_loss after that became increasing and that is cause overfitting , i think the problem with the way i extract the features from the images as I got negative values like that</p>
<pre><code class="lang-auto">
1.06231242e-01, -4.80975091e-01, -2.29298934e-01, -2.22610545e+00,
       -1.70528257e+00,  1.00546718e+00, -3.37958717e+00,  8.62520158e-01,
        1.09218419e+00,  1.02676439e+00, -5.19405723e-01, -1.97264120e-01,
        6.29661739e-01,  8.84239256e-01, -2.18081331e+00,  1.23091125e+00,
       -1.20287977e-01,  7.21974909e-01,  2.16648507e+00,  4.63137746e-01,
        4.28042859e-01, -7.97591209e-01, -7.74370611e-01, -4.94653493e-01,
        1.33503842e+00,  5.22300005e-01, -2.49824524e+00, -4.03152406e-01,
       -1.78791010e+00, -1.03124011e+00,  8.80928874e-01, -1.29657283e-01,
       -2.61071831e-01, -3.11868429e-01,  5.31454265e-01,  4.07434464e-01,
       -3.46822053e-01,  7.70217240e-01,  1.25887111e-01, -1.67335427e+00, ....
</code></pre>
<p>the code is</p>
<blockquote>
<p>class Identity(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
def forward(self, x):<br>
return x<br>
File = ‘file.pth.tar’<br>
model = create_model(My_model’,pretrained=False)<br>
model.load_state_dict(torch.load(File))<br>
model.head = Identity()<br>
model.eval()</p>
</blockquote>
<p>I found this way in this forum to freeze the last layer as I don’t need to make a classification</p>

ptrblck: <p>Your code wouldn’t freeze the last layer but replace it with an <code>Identity</code> layer, so that the output activations of the penultimate layer will be returned.<br>
If you don’t use an activation function such as <code>nn.ReLU</code> the activations could certainly be negative, so I’m unsure why negative values would be a concern.</p>

samm: <p>thanks a lot for this clarification another question if you please the model should have two methods which are forward_feature and forward . if I need to get the features from forward_feature using the <code>Identity</code> layer . I tried to implement like that</p>
<pre><code class="lang-auto">&gt; class Identity(nn. Module):
&gt;     def __init__(self):
&gt;         super().__init__()
&gt;     def forward_feature(self, x):
&gt;         return x
&gt; File = 'file.pth.tar'
&gt; model = create_model(model_name,pretrained=False)
&gt; model.load_state_dict(torch.load(File),strict=False)
&gt; model.head = Identity()
&gt; model.eval()
</code></pre>
<p>but i got</p>
<pre><code class="lang-auto">in forward
    out = self.head(x)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 175, in _forward_unimplemented
    raise NotImplementedError
NotImplementedError
</code></pre>
<p>I don’t know why it calls the <code>forward</code> method and I used the <code>forward_feature</code> method</p>

ptrblck: <p>If you pass the input to an instance of <code>nn.Module</code> via:</p>
<pre><code class="lang-python">output = model(input)
</code></pre>
<p>the <code>__call__</code> method will be called, will register hooks (and potentially perform other bookkeeping), and will then call <code>forward</code>, which is why you need to define this method.<br>
If you want to call a custom method, either call it in <code>forward</code> or directly via <code>output = model.forward_feature(input)</code>. The latter case will of course skip the hook registration etc. so don’t rely on these features.</p>

samm: <p>yes i get it , thanks but why every time i run the code i got different numbers for tensors of the images i try to print the tensor result and get different values every time . i used model.eval() but there is no sense in the code</p>

ptrblck: <p>I don’t know what might be causing it, so feel free to post a minimal, executable code snippet reproducing the issue.</p>

samm: <p>i will but just to make sure from this option <code>pretrained=False</code> if i have <code>.pth.tar</code> file i should set   <code>pretrained=False</code> and if i don’t i need to make it <code>True</code></p>

ptrblck: <p>Your code is unfortunately not executable, as neither the file nor all methods are defined.</p>

samm: <p>Do you mean you need the file of the model too ?</p>

ptrblck: <p>I would need a script, which I could copy/paste and execute locally to see and debug the issue.<br>
For this, the model definition would be needed as well as (random) inputs, which would show that the model outputs different values in <code>eval()</code> mode.</p>

samm: <p>Thanks but excuse me how can this be random and i load the pre-trained file with .pth.tar ? or i need to make other change in the code ?</p>

ptrblck: <p>I don’t know what’s causing the issue as I don’t have any code for debugging.</p>

samm: <p>can i upload the file here , please?</p>

ptrblck: <p>Check if the file is even needed or of a randomly initialized model with static inputs could also reproduce the issue. In the latter case, just post the executable code here, please.</p>

samm: <p>excuse me how can I check ?<br>
I tried to remove the path of the file and found it run with a random number<br>
if the model initialized the weights randomly , will it be a problem in training or using the model ?<br>
what is the best " the model initialized randomly " or using the file ?</p>

ptrblck: <p>If I understand the issue correctly, you are using a model, call <code>model.eval()</code>, feed static input into it, and are seeing different results.<br>
This is unexpected and we would need to have an executable code snippet in order to debug it.<br>
To do so, I don’t believe that loading a specific <code>state_dict</code> or the real dataset is necessary and would claim that the issue should also be visible when just initializing the model and feeding it with a static (random) input.<br>
Write a script, which creates a model instance, create a single random input tensor, call <code>model.eval()</code>, and run multiple forward passes. If the output differs by more than the known limited floating point precision, post the code here.</p>