Performing evaluation on the test set

Vishu_Gupta: <p>I have implemented the evaluation of the test set as follows:</p>
<pre><code class="lang-auto">n_epochs = 1000 
batch_size = 32 
loss_train=[]

for epoch in range(n_epochs):

    permutation1 = torch.randperm(trainX.size()[0])
    for i in range(0,trainX.size()[0], batch_size):

        optimizer.zero_grad()
        indices1 = permutation1[i:i+batch_size]
        batch_x_train, batch_y_train = trainX[indices1], trainY[indices1]

        model.train()
        outputs = model.forward(batch_x_train)
        train_loss = criterion(outputs,batch_y_train)
        train_loss.backward()
        optimizer.step()
        loss_train.append(train_loss.item())

    model.eval()
    y_pred = model(valX)
    val_loss = criterion(y_pred, valY)        

    avg_train_loss = sum(loss_train) / len(loss_train)
    print('epoch {},  train loss {},  val loss{}'.format(epoch, avg_train_loss, val_loss)) 

model.eval()
y_pred = model(testX)
test_loss = criterion(y_pred, testY) 
print('test loss is {}'.format(test_loss))
</code></pre>
<p>Is this the correct way to evaluate the model on the test set? Also, where and how should I save the model in this case ( torch.save() or model.state_dict() ) if in the future all I would want to do is to load the model and just use it on the test set?</p>

ptrblck: <p>Assuming <code>valX</code> is a tensor with the complete validation data, then this approach would be generally right, but you might of course run out of memory, if this tensor is too large.<br>
The usual approach would be to wrap it in a <code>Dataset</code> and <code>DataLoader</code> and get the predictions for each batch. The <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">data loading tutorial</a> gives you some information how to create a <code>Dataset</code> and <code>DataLoader</code>.</p>
<p>Also, to save memory during evaluation and test, you could wrap the validation and test code into a <code>with torch.no_grad()</code> block.</p>

Vishu_Gupta: <p>Do you mean to say that for evaluation and test set the code should be:</p>
<pre><code class="lang-auto">with torch.no_grad():
    model.eval()
    y_pred = model(valX)
    val_loss = criterion(y_pred, valY)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">with torch.no_grad():
    model.eval()
    y_pred = model(test)
    test_loss = criterion(y_pred, testY)
</code></pre>
<p>Also, how about the answer to the second part of the question where I ask about the best way to save the model?</p>

ptrblck: <p>The usage of <code>no_grad()</code> and <code>model.eval()</code> is correct and if you are not running our of memory using the complete datasets, your approach should work.</p>
<p>To store the model, you should save the <code>state_dict()</code> as described <a href="https://pytorch.org/docs/stable/notes/serialization.html">here</a>.</p>

Padmaksha_Roy: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I have a encoder decoder network where I am adding a softmax layer at the encoder. The training loss is converging nicely but when I run the evaluation on the same train data, it is giving very poor results. Can you check once plz if the test evaluation part is correct?</p>
<blockquote>
<pre><code>def forward(self, x):

    z = self.encode(x)
    logits = self.classifier(z)
    reconstruction = self.decode(z) 
    return logits, reconstruction
</code></pre>
</blockquote>
<blockquote>
<p>from torch.utils.data import Dataset, DataLoader</p>
</blockquote>
<blockquote>
<p>class DataBuilder(Dataset):</p>
</blockquote>
<blockquote>
<pre><code>def __init__(self, path):
    #self.x, self.standardizer, self.wine = load_data(DATA_PATH)
    self.x, self.y = load_data(DATA_PATH)
    self.len=self.x.shape[0]
</code></pre>
</blockquote>
<blockquote>
<pre><code>def __getitem__(self,index):      
    return self.x[index], self.y[index]
def __len__(self):
    return self.len
</code></pre>
</blockquote>
<blockquote>
<p>data_set=DataBuilder(DATA_PATH)</p>
<p>trainloader=DataLoader(dataset=data_set,batch_size=1)</p>
</blockquote>
<blockquote>
<p>accs = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>with torch.no_grad():</p>
</blockquote>
<blockquote>
<p>for batch_idx, data in enumerate(trainloader):</p>
<pre><code>    x_data = data[0]
    y_data = data[1]

    recon_batch, logits_batch = model(x_data)
    acc = (logits_batch.round() == y_data).float().mean()
    accs.append(acc)
</code></pre>
<p>print(“%.2f%% (+/- %.2f%%)” % (np.mean(accs)*100, np.std(accs)*100))</p>
</blockquote>

ptrblck: <p>I’m not familiar with your use case and thus don’t know if the accuracy calculation makes sense or not. Check the raw outputs before rounding and compare some of these outputs manually to their targets to understand what your model is trying to predict.</p>

Padmaksha_Roy: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks for your reply. I found out that my issue is with the architecture itself and not inference. I have a simple encoder-decoder model and I am trying to add a softmax classifier layer from the encoder so that I can optimize the classification and reconstruction loss jointly.</p>
<blockquote>
<p>class Autoencoder(nn.Module):<br>
def <strong>init</strong>(self,D_in,H=50,H2=12,latent_dim=7):</p>
<pre><code>    #Encoder
    super(Autoencoder,self).__init__()
    self.linear1=nn.Linear(D_in,H)
    self.lin_bn1 = nn.BatchNorm1d(num_features=H)
    self.linear2=nn.Linear(H,H2)
    self.lin_bn2 = nn.BatchNorm1d(num_features=H2)
    self.linear3=nn.Linear(H2,H2)
    self.lin_bn3 = nn.BatchNorm1d(num_features=H2)
    self.num_class = 3

    self.fc1 = nn.Linear(H2, latent_dim)
</code></pre>
</blockquote>
<blockquote>
<pre><code>    self.fc3 = nn.Linear(latent_dim, latent_dim)
    self.fc4 = nn.Linear(latent_dim, H2)
</code></pre>
</blockquote>
<blockquote>
<h1># Decoder</h1>
<pre><code>    self.linear4=nn.Linear(H2,H2)
    self.lin_bn4 = nn.BatchNorm1d(num_features=H2)
    self.linear5=nn.Linear(H2,H)
    self.lin_bn5 = nn.BatchNorm1d(num_features=H)
    self.linear6=nn.Linear(H,D_in)
    self.lin_bn6 = nn.BatchNorm1d(num_features=D_in)

    self.relu = nn.ReLU()
    self.classifier = nn.Linear(latent_dim, self.num_class)
   # self.classifier = nn.Softmax()

def encode(self, x):
    lin1 = self.relu(self.lin_bn1(self.linear1(x)))
    lin2 = self.relu(self.lin_bn2(self.linear2(lin1)))
    lin3 = self.relu(self.lin_bn3(self.linear3(lin2)))


    fc1 = F.relu(self.fc1(lin3))
    return fc1
</code></pre>
</blockquote>
<blockquote>
<pre><code>def decode(self, z):
    fc3 = self.relu(self.fc3(z))
    fc4 = self.relu(self.fc4(fc3))#.view(128, -1)

    lin4 = self.relu(self.lin_bn4(self.linear4(fc4)))
    lin5 = self.relu(self.lin_bn5(self.linear5(lin4)))
    return self.lin_bn6(self.linear6(lin5))



def forward(self, x):
    # mu, logvar = self.encode(x)
    z = self.encode(x)
    logits = self.classifier(z)
    reconstruction = self.decode(z)
    #z = torch.vstack(z)
    #print ("Z shape:", z.shape)
    #print ("Z shape:", type(z))
    # self.decode(z) ist später recon_batch, mu ist mu und logvar ist logvar
    #return self.decode(z), mu, logvar
    return logits, reconstruction
</code></pre>
</blockquote>
<blockquote>
<p>class customLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(customLoss, self).<strong>init</strong>()<br>
self.mse_loss = nn.MSELoss(reduction=“sum”)<br>
self.classification_criterion = nn.CrossEntropyLoss()</p>
</blockquote>
<blockquote>
<pre><code># x_recon ist der im forward im Model erstellte recon_batch, x ist der originale x Batch, mu ist mu und logvar ist logvar
#def forward(self, x_recon, x, mu, logvar):
def forward(self, x_recon, x, logits, targets):
    loss_MSE = self.mse_loss(x_recon, x)
    
    #loss_KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    classification_loss = self.classification_criterion(logits, torch.max(targets, 1)[1])

    return loss_MSE + classification_loss
</code></pre>
</blockquote>
<blockquote>
<p>def train(epoch):<br>
model.train()<br>
train_loss = 0<br>
for batch_idx, data in enumerate(trainloader):<br>
<a class="hashtag-cooked" href="/c/data/37" data-type="category" data-slug="data" data-id="37"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>data</span></a> = data.to(device)<br>
<span class="hashtag-raw">#print</span> (“data:”, data)<br>
x_train = data[0]<br>
y_train = data[1]</p>
<pre><code>   # print ("x train:", x_train.shape)
    print ("y train:", y_train)


    optimizer.zero_grad()
    # recon_batch, mu, logvar = model(data)
    # loss = loss_mse(recon_batch, data, mu, logvar)

    logits,recon_batch  = model(x_train)
    print ("logits train:", logits)

    _, predicted = torch.max(logits.data, 1)
    print ("predicted:", predicted)

   # print ("logits shape:", logits.shape)
   # print ("recon_batch shape:", recon_batch.shape)
    loss = loss_mse(recon_batch, x_train, logits, y_train)

    loss.backward()
    train_loss += loss.item()
    optimizer.step()
</code></pre>
</blockquote>
<blockquote>
<pre><code>if epoch % 200 == 0:
    print('====&gt; Epoch: {} Average loss: {:.4f}'.format(
        epoch, train_loss / len(trainloader.dataset)))
    train_losses.append(train_loss / len(trainloader.dataset))
</code></pre>
</blockquote>
<p>When I print it, the outputs of CE loss are not probabilities. And I see the prediction always as the same class(total classes are 3).</p>
<blockquote>
<p>logits train: tensor([[ 0.4869, -0.6447, -0.6673],<br>
[ 0.8050, -1.0218, -0.6015],<br>
[ 0.4048, -0.5664, -0.5431],<br>
[ 0.8462, -1.0309, -0.6237]], grad_fn=)<br>
predicted: tensor([0, 0, 0, 0])<br>
y train: tensor([[0],<br>
[2],<br>
[2],<br>
[1]])<br>
logits train: tensor([[ 0.4088, -0.5317, -0.2215],<br>
[ 0.7658, -0.9260, -0.5844],<br>
[ 0.7539, -0.9134, -0.5994],<br>
[ 0.4659, -0.6436, -0.7550]], grad_fn=)<br>
predicted: tensor([0, 0, 0, 0])<br>
y train: tensor([[0],<br>
[0],<br>
[1],<br>
[1]])</p>
</blockquote>
<p>Can you please tell me what am I doing wrong here. Expected o/ps should be probabilities and also why it is predicting the same class always? Thank you so much!</p>

ptrblck: <aside class="quote no-group" data-username="Padmaksha_Roy" data-post="7" data-topic="85137">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/padmaksha_roy/48/47934_2.png" class="avatar"> Padmaksha_Roy:</div>
<blockquote>
<p>Expected o/ps should be probabilities</p>
</blockquote>
</aside>
<p>That’s not the case since <code>nn.CrossEntropyLoss</code> expects raw logits (which are unbound) and will apply <code>F.log_softmax</code> internally.</p>
<aside class="quote no-group" data-username="Padmaksha_Roy" data-post="7" data-topic="85137">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/padmaksha_roy/48/47934_2.png" class="avatar"> Padmaksha_Roy:</div>
<blockquote>
<p>and also why it is predicting the same class always?</p>
</blockquote>
</aside>
<p>Your model seems to be overfitting to <code>class0</code> so check if this is the majority class and use a weighted sampling or weighted loss if that’s the case to counter this effect.</p>

Padmaksha_Roy: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for the reply. I could solve the issue.</p>