Losses and Accuracies are zero after first epoch

DL_jain: <p>Hey,</p>
<p>I am using PyTorch for <strong>multiclass classification</strong>. I’ve one hot encoded the labels. On running the model I am getting train and validation losses and accuracies for first epoch but for all the next epochs their value is coming out to be 0. Also I’m not using a dataloader, instead I’m using a function of my own to load the data in batches. I’m a beginner please guide me.</p>
<p>def train_model(model, criterion, optimizer, num_epochs=10):</p>
<p>for epoch in range(num_epochs):<br>
print(‘Epoch {}/{}’.format(epoch+1, num_epochs))<br>
print(’-’ * 10)</p>
<pre><code>   for phase in ['train', 'val']:
        if phase == 'train':
            model.train()
        else:
            model.eval()
        running_loss = 0.0
        running_corrects = 0.0
        for i,(inputs, labels) in enumerate(dataloaders[phase]):
            inputs = inputs.to(device)
            labels = labels.to(device)
            outputs = model(inputs)
            _,lab1 = torch.max(labels.data, 1)
            loss = criterion(outputs, labels)
            if phase == 'train':
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

            _, preds = torch.max(outputs, 1)
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == lab1)

        epoch_loss = running_loss / dataset_sizes[phase]
        epoch_acc = float(running_corrects/dataset_sizes[phase])

        print('{} loss: {:.4f}, acc: {:.4f}'.format(phase,
                                                    epoch_loss,
                                                    epoch_acc))
return model
device = torch.device("cpu")

model_ft = models.resnet50(pretrained=True).to(device)

for param in model_ft.parameters():
param.requires_grad = False

model_ft.fc = nn.Sequential(
    nn.Linear(2048, 1000),
     nn.ReLU(inplace=True),
     nn.Linear(1000, 136)).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model_ft.fc.parameters(), lr=0.001)    

model_ft = train_model(model_ft, criterion, optimizer, num_epochs=10)</code></pre>

ptrblck: <p>One-hot encoded targets wouldn’t work with <code>nn.CrossEntropyLoss</code>, so I’m not sure why</p>
<pre><code class="lang-python">loss = criterion(outputs, labels)
</code></pre>
<p>doesn’t raise an error.<br>
Could you print the shapes of both tensors before passing them to the criterion?</p>
<p>Also, check the shapes and values of <code>epoch_loss</code> and <code>epoch_acc</code> to avoid accidental broadcasting etc.</p>
<p>How did you define <code>dataloaders</code>, if you are not using the built-in <code>DataLoader</code> class?</p>

DL_jain: <p>Thanks for the reply!<br>
I’m using</p>
<blockquote>
<p>loss = criterion(outputs, lab1)</p>
</blockquote>
<p>not (outputs, labels). Sorry for the typo.</p>
<p>outputs = torch.Size([16,136])<br>
lab1 = torch.Size([16])</p>
<p>epoch_loss = 43449.26647949219<br>
epoch_acc.shape = torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
train Loss: 3.4776 Acc: 0.1683</p>
<p>epoch_loss = 30073.398727416992<br>
epoch_acc.shape = torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
val Loss: 3.3855 Acc: 0.1740</p>
<p>I’m using the following function for dataloading:</p>
<pre><code class="lang-auto">def image_generator(file, label_file, batch_size = 16):
    new_bs=0
    size = len(file)
    for i in range(math.floor(size/batch_size)):
        batch_x = file[new_bs:(new_bs+batch_size)]
        batch_y = label_file[new_bs:(new_bs+batch_size)]
        new_bs += batch_size
        yield(batch_x, batch_y)
</code></pre>
<pre><code class="lang-auto">dataloaders = {'train': image_generator(X_train, y_train, batch_size = 16), 
                 'val': image_generator(X_val, y_val, batch_size = 16)
                }
</code></pre>
<p>Here in image_generator function, X_train is a tensor and y_train is a NumPy array.<br>
If there’s any other better method for dataloading please suggest, as I have all the images in single folder and I can’t use ImageFolder.</p>

Ishal_Garg: <p>Hy DL_jain!</p>
<p>Loss is calculated between original labels and your output probability. So your loss should be<br>
<code>loss = criterion(output,labels)</code> as mentioned by <strong>ptrblck</strong></p>

DL_jain: <p>Hey, thanks for the reply!<br>
I’m getting this error on running <code>loss = criterion(output,labels)</code>:<br>
`</p>
<blockquote>
<p>RuntimeError: 1D target tensor expected, multi-target not supported</p>
</blockquote>
<p>Also <a class="mention" href="/u/ptrblck">@ptrblck</a> is right in saying in that  <code>loss = criterion(output, labels)</code> will not work since labels are one-hot encoded. This is because the target in CrossEntropyLoss only accepts 1D tensor.</p>

ptrblck: <p>Thanks for the update.<br>
Could you print the <code>preds</code> and <code>lab1</code> tensors for the second epoch, which yields a zero accuracy and zero loss?<br>
It would be interesting to see, how these values both can be calculated using the current code.</p>

zains1234: <p>Can you help me <a class="mention" href="/u/ptrblck">@ptrblck</a>, my program generate zero loss to in first epoch</p>
<pre><code class="lang-auto">def train_dataset(train_loader, val_loader, drivePath, type):
    lr=1e-2
    num_epochs=5

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 
    model = None
    if (type == 'custom'):
        model = VGG16_CUSTOM_NET()
    else:
        model = VGG16_ORI_NET() 
    print("[INFO] reset the weight before starting")
    model.apply(reset_weights)
    model = model.to(device=device)
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr= lr) 

    for epoch in range(num_epochs): #I decided to train the model for 50 epochs
        loss_var = 0
        
        for idx, (images, labels) in enumerate(train_loader):
            images = images.to(device=device)
            labels = labels.to(device=device)
            ## Forward Pass
            optimizer.zero_grad()
            scores = model(images)
            loss = criterion(scores,labels)
            loss.backward()
            optimizer.step()
            loss_var += loss.item()
            print(f'Epoch [{epoch+1}/{num_epochs}] || Step [{idx+1}/{len(train_loader)}] || Loss:{loss_var/len(train_loader)} || Loss Item:{loss.item()}')
        print(f"Loss at epoch {epoch+1} || {loss_var/len(train_loader)}")

        with torch.no_grad():
            correct = 0
            samples = 0
            for idx, (images, labels) in enumerate(val_loader):
                images = images.to(device=device)
                labels = labels.to(device=device)
                outputs = model(images)
                _, preds = outputs.max(1)
                correct += (preds == labels).sum()
                samples += preds.size(0)
            print(f"accuracy {float(correct) / float(samples) * 100:.2f} percentage || Correct {correct} out of {samples} samples")
</code></pre>
<pre><code class="lang-auto">class VGG16_ORI_NET(nn.Module):
    def __init__(self):
        super(VGG16_ORI_NET, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1)

        self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, padding=1)
        self.conv4 = nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, padding=1)

        self.conv5 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, padding=1)
        self.conv6 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1)
        self.conv7 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1)

        self.conv8 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, padding=1)
        self.conv9 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1)
        self.conv10 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1)

        self.conv11 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1)

        self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)

        self.fc14 = nn.Linear(25088, 4096)
        self.fc15 = nn.Linear(4096, 4096)
        self.fc16 = nn.Linear(4096, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = self.maxpool(x)
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        x = self.maxpool(x)
        x = F.relu(self.conv5(x))
        x = F.relu(self.conv6(x))
        x = F.relu(self.conv7(x))
        x = self.maxpool(x)
        x = F.relu(self.conv8(x))
        x = F.relu(self.conv9(x))
        x = F.relu(self.conv10(x))
        x = self.maxpool(x)
        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x))
        x = self.maxpool(x)
        x = x.reshape(x.shape[0], -1)
        x = F.relu(self.fc14(x))
        x = F.dropout(x, 0.5) #dropout was included to combat overfitting
        x = F.relu(self.fc15(x))
        x = F.dropout(x, 0.5)
        x = self.fc16(x)
        return x
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/209e17679f68de06f7c8b38952e93f9d8b07ab43.png" data-download-href="https://discuss.pytorch.org/uploads/default/209e17679f68de06f7c8b38952e93f9d8b07ab43" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/209e17679f68de06f7c8b38952e93f9d8b07ab43.png" alt="image" data-base62-sha1="4ExZhih0U8UOD94KbaqpvEWqHPJ" width="690" height="249" data-dominant-color="383838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">835×302 9.36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If you are concerned about the zero loss during training, check the output of your model and the corresponding labels to see if the zero loss is expected. Maybe your model is overfitting to a single class and the current batch labels contain only this particular class.</p>

zains1234: <p>Thank you for your reply.<br>
I will check my model output and the corresponding labels</p>