Static model outputs when using a saved network

Revan: <p>Hi all,</p>
<p>Sorry if this seems trivial, but this problem is really throwing my limited experience with pytorch for a loop. I have a binary classification model that I trained in a notebook, which seems to be fairly accurate (~89%) with an ROC AUC of 0.95.  The dataset used has 100,000 entries, half of which have a label of 1, the other half of which have a label of 0. Despite the model’s success when being trained and tested in the notebook, when I save the model’s state to a .pth after finishing training and load the .pth in a different python script, the fails, outputting the same values constantly despite shuffling in the entries. I’ve attached screenshots of the outputs to show what I mean:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/5191fd949f5b1ca6b317a953e223dd30d55206a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/5191fd949f5b1ca6b317a953e223dd30d55206a5" title="PNG image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/5191fd949f5b1ca6b317a953e223dd30d55206a5.png" alt="PNG image" data-base62-sha1="bDBsq9azQdqwAiuEi8dCTeArwUJ" width="690" height="145" data-dominant-color="2E343B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PNG image</span><span class="informations">871×184 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when running a separate time:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac9b146e8e7d4ef0ee73be49d34fe7ea111b2546.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac9b146e8e7d4ef0ee73be49d34fe7ea111b2546" title="PNG image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac9b146e8e7d4ef0ee73be49d34fe7ea111b2546.png" alt="PNG image" data-base62-sha1="oCWnFXaRNSDe1y59XIDbIysVKKO" width="690" height="139" data-dominant-color="2F343C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PNG image</span><span class="informations">873×177 12.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I checked the weights on both sides and they seem to be equivalent as well, leading me to believe an export error isn’t the cause. Does anyone have any idea why this is happening and how I can fix this?</p>
<p>Thanks in advance.</p>
<p>I’ve displayed the script I’m using here:</p>
<pre><code class="lang-auto">import torch
import pandas as pd
import numpy
from sklearn.preprocessing import StandardScaler
import joblib


model = torch.nn.Sequential(
    torch.nn.Linear(8, 15),
    torch.nn.ReLU(),
    torch.nn.Linear(15, 20),
    torch.nn.ReLU(),
    torch.nn.Dropout(p=0.5),
    torch.nn.Linear(20, 1),
    torch.nn.Sigmoid()
)
model.load_state_dict(torch.load("model.pth"))


df = pd.read_csv("out.csv", index_col=0)
df = df.drop(columns=['Transverse Momentum', 'Delta Angle'])
data = df.iloc[:, :-1].values
df = df.sample(frac=1).reset_index(drop=True)  
scaler = joblib.load('scaler.pkl')
data_scaled = scaler.fit_transform(data)
inputs = torch.tensor(data_scaled, dtype=torch.float32)  

model.eval()  
    
with torch.no_grad():  
    outputs = model(inputs).squeeze()

output_df = pd.DataFrame(outputs.numpy(), columns=["Model Output"])
result_df = pd.concat([df.iloc[:, :-1], output_df], axis=1) 
print(result_df)
</code></pre>

ptrblck: <p>Compere the model outputs using static inputs (e.g. via <code>torch.zeros</code>). If the output matches in eval mode the <code>state_dict</code> was properly restored. If so, check the data processing next as the issue might come from this part of the code.</p>

Revan: <p>Thank you for your reply! I’m not sure if I understood completely correctly, but I added the following line of code</p>
<pre><code class="lang-auto">inputs[inputs!=0] = 0

</code></pre>
<p>before I fed data into the model, and all the outputs are returning as the same:</p>
<pre><code class="lang-auto">                Px          Py           Pz            E          aPx         aPy          aPz           aE  Label  Model Output
0       498.952456  942.892961  4086.036224  4301.113063   249.016418  662.537552  2003.704279  2125.040240    1.0      0.123355
1       -33.194958 -114.358111 -1272.467161  1335.982566    17.217558 -189.981795  -330.377740   381.495711    0.0      0.123355
2        92.637850  -52.984615   266.590605   384.930240    33.902684  -13.347508   198.809090   202.120267    1.0      0.123355
3      -284.050487  346.318370   343.771929   811.319527   -24.833545  -80.035296   -57.113204   101.411398    0.0      0.123355
4        19.007433   76.927001    68.024735   720.459885    32.253253   25.372707    75.189907    85.659609    0.0      0.123355
...            ...         ...          ...          ...          ...         ...          ...          ...    ...           ...
99995   190.489468 -543.171435  -149.002496   742.264802   122.097355 -501.058770   -78.169397   521.611071    0.0      0.123355
99996  -751.130983  373.254110  -148.271525  1026.438270  -240.812983   79.982750  -245.473745   353.051686    1.0      0.123355
99997  -183.398868  -97.472225   280.874511   694.839428   119.017551   -1.913719   203.885222   236.089016    0.0      0.123355
99998   127.495375  -95.355002    81.195261   496.588585   164.695552   44.174902  -120.185642   208.616000    0.0      0.123355
99999  1776.908356 -596.724914 -2547.084304  3268.601361  1735.300013 -651.433658 -2464.186455  3083.479662    1.0      0.123355
</code></pre>
<p>As for data processing, I just used sklearn’s StandardScaler() when training, saved the scaler using joblib, and used the same scaler in the python script. Anything else I might be missing?</p>
<p>Here’s the code I used when training the model, specific to the data processing:</p>
<pre><code class="lang-auto">df = pd.read_csv("out.csv", index_col=0)
df = df.drop(columns=['Transverse Momentum', 'Delta Angle'])
df = df.sample(frac=1).reset_index(drop=True)  
print(df)

data = df.iloc[:, :-1].values
labels = df.iloc[:, -1].values


scaler = StandardScaler()
data_scaled = scaler.fit_transform(data)
joblib.dump(scaler, 'scaler.pkl')


tensorData = []
for i in range(len(data_scaled)):
    tensorData.append(torch.FloatTensor([float(j) for j in data_scaled[i]]))
data_train = torch.stack(tensorData[0:2000])
labels_train = torch.FloatTensor(labels[0:2000])

data_test = torch.stack(tensorData[2000:])
labels_test = torch.FloatTensor(labels[2000:])


train_dataset = torch.utils.data.TensorDataset(data_train, labels_train)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=5, shuffle=True)
</code></pre>

ptrblck: <p>I assume the static output is equal for both models (i.e. before saving the <code>state_dict</code> and after restoring it)? If so, try to use a random tensor via <code>torch.randn</code> next and compare the outputs as well.</p>