Binary classification neural network always predicts zero

username: Sena_Korkut
cooked: <p>Hi, I am new at Pytorch and I am trying to classify Ocular Dataset with  both NN and CNN. However, although my loss is decreasing through epoch, my accuracy stays same, which is because my model always predicts class zero. I have searched for a reason but nothing fixed my issue. I already have gray scaled and normalized and flatten the dataset, I shuffled before splitting the dataset into train, validation and test. I also shuffled while loadşng the data. And lastly, I tried to change batch size, learning rate and epoch size but it did not work. Can you please help me to find the problem?</p>
<p>I use BCEWithLegitsLoss for my loss function and SGD for optimizer.</p>
<p>The code is below for getting dataset:</p>
<blockquote></blockquote>
<pre><code>def get_dataset(root):
#os.mkdir('/content/gdrive/My Drive/Colab Notebooks/images/1')
#os.mkdir('/content/gdrive/My Drive/Colab Notebooks/images/0')
label_data = pd.read_excel('/content/gdrive/My Drive/Colab Notebooks/labels.xlsx')
labels = label_data.iloc[:,1]
ids = label_data.iloc[:,0]
#idx = 0
#for id in ids: 
#os.rename('/content/gdrive/My Drive/Colab Notebooks/images/'+ str(id) +'.jpg', '/content/gdrive/My Drive/Colab Notebooks/images/'+ str(labels[idx]) +'/'+ str(id) + '.jpg')
transform_data = T.Compose([T.Grayscale(), T.ToTensor()  ])
train_dataset = ImageFolder(root=root,transform=transform_data)
images = []
labels = []
#check = 0
for image, label in iter(train_dataset):
    #print(" " + str(check), end='')
    T.Normalize(2, 0.5)(image)
    images.append(image.view(32768))
    labels.append(label)
labels = torch.tensor(labels)
images = torch.stack(images)
print("Before shuffle")
#Shuffle data
l = list(range(len(ids)))
random.shuffle(l)
train_shuffled = l[0:images.shape[0] * 5 // 7]
val_shuffled = l[len(train_shuffled):len(train_shuffled) + images.shape[0] // 7]
test_shuffled = l[len(train_shuffled) + images.shape[0] // 7:len(train_shuffled) + images.shape[0] // 7 
                        +len(train_shuffled) + images.shape[0] // 7]
# Construct training, validation and test sets
train_data = images[train_shuffled, ...]
train_label = labels[train_shuffled]
val_data = images[val_shuffled, ...]
val_label = labels[val_shuffled]
test_data = images[test_shuffled, ...]
test_label = labels[test_shuffled]
train_dataset = OcularDataset(data = train_data, labels = train_label)
val_dataset = OcularDataset(data = val_data, labels = val_label)
test_dataset = OcularDataset(data = test_data, labels = test_label)
return train_dataset, val_dataset, test_dataset
</code></pre>
<p>And this is my main, train and test functions:</p>
<blockquote>
<p>max_epoch = 50<br>
train_batch = 32<br>
test_batch = 32<br>
learning_rate = 0.0001<br>
device = torch.device(‘cuda’)<br>
if(torch.cuda.is_available()):<br>
print(“Using GPU”)</p>
</blockquote>
<blockquote>
<p>def main(): # you are free to change parameters<br>
if(torch.cuda.is_available()):<br>
print(“Using GPU”)<br>
# Create train dataset loader<br>
train_loader = DataLoader(train_set, batch_size=train_batch, shuffle=True)<br>
# Create validation dataset loader<br>
valid_loader = DataLoader(val_set, batch_size=test_batch, shuffle = True)<br>
# Create test dataset loader<br>
test_loader = DataLoader(test_set, batch_size=test_batch, shuffle = True)<br>
# initialize your GENet neural network<br>
model = FNet().to(device)<br>
# define your loss function<br>
loss_func = nn.BCEWithLogitsLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=5e-04) # you can play with momentum and weight_decay parameters as wel<br>
train_acc = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
train_loss = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
valid_acc = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
valid_loss = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
min_loss = 10000<br>
# start training<br>
# for each epoch calculate validation performance<br>
# save best model according to validation performance<br>
for epoch in range(max_epoch):<br>
train(epoch, model, loss_func, optimizer, train_loader, train_acc, train_loss)<br>
acc,loss = test(model, valid_loader, loss_func)<br>
print()<br>
valid_acc.append(acc)<br>
valid_loss.append(loss)<br>
if loss &lt; min_loss:<br>
min_loss = loss<br>
torch.save(model, ‘best_path.pth’)<br>
return (train_acc, train_loss, valid_acc, valid_loss)<br>
‘’’ Train your network for a one epoch ‘’’</p>
</blockquote>
<blockquote>
<p>def train(epoch, model, loss_func, optimizer, loader,train_acc,train_loss): # you are free to change parameters<br>
model.train()<br>
loss_tot = 0<br>
acc_tot = 0<br>
prediction_tot = 0<br>
for batch_idx, (data, labels) in enumerate(loader):<br>
start = time.time()<br>
# TODO:<br>
# Implement training code for a one iteration<br>
data = data.to(device=device, dtype=torch.float32)<br>
labels = labels.to(device=device,  dtype=torch.float32)<br>
optimizer.zero_grad()<br>
output = model(data)<br>
loss = loss_func(output, labels.float().unsqueeze(1))<br>
loss.backward()<br>
optimizer.step()<br>
end = time.time()<br>
predictions = output.argmax(dim=1)<br>
corrects = 0<br>
for idx in range(labels.shape[0]):<br>
if predictions[idx] == labels[idx]:<br>
corrects += 1<br>
<span class="hashtag-raw">#corrects</span> = (predictions == labels).sum()<br>
acc = corrects / labels.shape[0]<br>
prediction_tot += labels.shape[0]<br>
acc_tot += acc<br>
loss_tot += loss.item()<br>
train_acc.append(acc)<br>
train_loss.append(loss)<br>
print(‘Epoch: [{0}][{1}/{2}]\t’<br>
‘Time {batch_time:.3f}\t’<br>
‘Loss {loss:.4f} ({loss_avg:.4f})\t’<br>
‘Accu {acc:.4f} ({acc_avg:.4f})\t’.format(<br>
epoch + 1, batch_idx + 1, len(loader),<br>
batch_time=end - start,<br>
loss=loss.item(), loss_avg = loss_tot / (batch_idx+1),<br>
acc = acc, acc_avg = acc_tot/(batch_idx+1)))<br>
‘’’ Test&amp;Validate your network ‘’’</p>
</blockquote>
<blockquote>
<p>def test(model, loader, loss_func): # you are free to change parameters<br>
model.eval()<br>
corrects = 0<br>
samples = 0<br>
with torch.no_grad():<br>
for batch_idx, (data, labels) in enumerate(loader):<br>
# TODO:<br>
# Implement test code<br>
start = time.time()<br>
data = data.to(device=device)  # move to device, e.g. GPU<br>
labels = labels.to(device=device)<br>
output = model(data)<br>
loss = loss_func(output, labels.float().unsqueeze(1))<br>
end = time.time()<br>
predictions = output.argmax(dim=1)<br>
for idx in range(labels.shape[0]):<br>
if predictions[idx] == labels[idx]:<br>
corrects += 1<br>
samples += (labels.shape[0])<br>
acc_avg = corrects/samples<br>
print(‘Time: {batch_time:.3f}\t’<br>
‘Loss: {loss:.4f}\t’<br>
‘Accu {acc_avg:.4f}\t’.format(<br>
batch_time=end-start,<br>
loss = loss.item(),<br>
acc_avg=acc_avg))<br>
return acc_avg, loss</p>
</blockquote>
<p>My FNet class is as follows:</p>
<blockquote>
<p>Blockquote<br>
class FNet(nn.Module):<br>
def <strong>init</strong>(self, **kwargs):<br>
super(FNet, self).<strong>init</strong>()<br>
self.hidden_layer1 = nn.Linear(32768, 1024)<br>
self.hidden_layer2 = nn.Linear(1024,256)<br>
self.hidden_layer3 = nn.Linear(256,32)<br>
self.output_layer = nn.Linear(32,1)</p>
</blockquote>
<pre><code>def forward(self, X): 
  x = F.relu(self.hidden_layer1(X))            
  x = F.relu(self.hidden_layer2(x)) 
  x = F.relu(self.hidden_layer3(x))          
  x = self.output_layer(x)
  x = torch.sigmoid(x)       
  return x
</code></pre>
<p>And finally, these are the plots for validation accuracy and loss in each epoch<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92e25ec3d0f44a7f2be477c459ce20cd8ec5cbb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/92e25ec3d0f44a7f2be477c459ce20cd8ec5cbb9" title="valid"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92e25ec3d0f44a7f2be477c459ce20cd8ec5cbb9_2_690x380.png" alt="valid" data-base62-sha1="kXoHNYyzmCZj3ZLyNGNXektNKC5" width="690" height="380" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92e25ec3d0f44a7f2be477c459ce20cd8ec5cbb9_2_690x380.png, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92e25ec3d0f44a7f2be477c459ce20cd8ec5cbb9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92e25ec3d0f44a7f2be477c459ce20cd8ec5cbb9.png 2x" data-dominant-color="FBF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">valid</span><span class="informations">770×425 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Kushaj
cooked: <p>When you use <code>nn.BCEWithLogLogits</code> it internally applies a sigmoid. So no need to apply sigmoid in your forward step (using sigmoid in your forward is just creating possible numerical instability).</p>
<p>In binary classification the output is a single neuron, and taking argmax of it does not make sense. It will always output the index 0. Instead for output you should do an if condition <code>if output[i]&gt;0.5</code> then it belongs to class 1 else it belongs to class 0.</p>
<p>Also, you would have to manually add sigmoid at test time as you would have removed it from the forward step.</p>