Pandas to Tensor fed to nn, model parameters not updating

username: Francisco_Obando
cooked: <p>Hello, I’m starting to use PyTorch and want to implement a simple working example of a binary classification. I have some data that is pre-processed with pandas, then transform the training and testing sets to tensors. I create a class with 2 hidden layers with ReLU activation functions, and then a single output (logit). I define the optimizer to be Adam with the model parameters, and the loss function BCEWithLogitLoss. When I create the instance the parameters are uniformly distributed, but when I do the training process the loss is not decreased, and all model parameters remain the same. There is no missing data on the tensors, and I made sure that with the untrained instance of the network the architecture would give a single output as expected.</p>
<p>I suspect the error comes from the loss.backward() or optimizer.step() steps, here is a sample of the data, and the code</p>
<pre><code class="lang-auto"># Define tensors to train the model
x_train_t = torch.tensor(x_train.values, dtype=torch.float)
x_test_t = torch.tensor(x_test.values, dtype=torch.float)
y_train_t = torch.tensor(y_train.values, dtype=torch.float)
y_test_t = torch.tensor(y_test.values, dtype=torch.float)


class NetClassifier(nn.Module):
    def __init__(self,hidden1,hidden2):
        super(NetClassifier,self).__init__()
        self.model = nn.Sequential(
            nn.Linear(13,hidden1),
            nn.ReLU(),
            nn.Linear(hidden1,hidden2),
            nn.ReLU(),
            nn.Linear(hidden2,1)
        )
    def forward(self,x):
        x = self.model(x)
        return x

# Create an instance of the class
net = NetClassifier(50,50)
print(net)


# Create DataSet and DataLoaders to train the neural network
EPOCHS = 5
BATCH_SIZE=32
LEARNING_RATE = 0.01

train_data_set = TensorDataset(x_train_t,y_train_t) 
test_data_set = TensorDataset(x_test_t,y_test_t)

train_dataloader = DataLoader(train_data_set,batch_size = BATCH_SIZE)
test_dataloader = DataLoader(test_data_set,batch_size = BATCH_SIZE)


# Define optimizer and loss function
optimizer = optim.Adam(net.parameters(),lr=LEARNING_RATE)
loss_function = nn.BCEWithLogitsLoss()


for epoch in range(EPOCHS):
    epoch_loss=0
    epoch_acc=0
    for xb,yb in train_dataloader:
        # Zero gradients for training
        optimizer.zero_grad()
        # Use current model parameters to predict output
        y_pred = net(xb)
        # Turn probabilities into prediction 
        pred = torch.round(torch.sigmoid(torch.flatten(y_pred)))
        # Calculate loss, use float type to calculate loss
        loss = loss_function(yb,pred)
        # Backpropagate
        loss.backward()
        # Step in the optimizer
        optimizer.step()
        epoch_loss+=loss.item()
        epoch_acc+= (yb == pred).float().mean()
# Print epoch loss
    print("Epoch {:&gt;02d} | Loss {:.5f} | Acc {:.3f}".format(epoch,epoch_loss/len(train_dataloader),epoch_acc/len(train_dataloader)))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da52ca0d1853a167256e1a5924ba43e7227cac46.png" alt="image" data-base62-sha1="v9nsKASF5oUz1mJTaYg65csYf9Y" width="663" height="231"></p>

username: AlphaBetaGamma96
cooked: <p>Hi!</p>
<p>I think the issue is with this line,</p>
<aside class="quote no-group" data-username="Francisco_Obando" data-post="1" data-topic="120772">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/francisco_obando/48/38093_2.png" class="avatar"> Francisco_Obando:</div>
<blockquote>
<pre><code class="lang-auto">        pred = torch.round(torch.sigmoid(torch.flatten(y_pred)))
</code></pre>
</blockquote>
</aside>
<p>I think the backward method for <code>torch.round</code> is <code>0</code>  with respect to all inputs. So, for example,</p>
<pre><code class="lang-auto">x=torch.randn(10, requires_grad=True) #returns tensor([-0.0534,  1.1239, ...])
y=torch.round(x)                      #round these values
y.backward(torch.ones_like(y))        #calculate gradient of round(x) with respect to x
x.grad                                #returns tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>And, given that this line happens after your network calculation, all the gradients of your weights/biases of your network are dependent on this line and therefore will have zero gradient! Perhaps pass in <code>torch.sigmoid(torch.flatten(y_pred))</code> rather than the rounded value, and see if that works! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Francisco_Obando
cooked: <p>Thank you for the feedback, there was a problem with using BCEWithLogitsLoss with only one output on the neural network and using the loss fuction with a sigmoid (since BCEwithLogitsLoss has the sigmoid function it was actually performing a sigmoid of a sigmoid to calculate the negative log likelihood). Thankfully the community has plenty of examples so I was able to get to the bottom of the issue.</p>

username: AlphaBetaGamma96
cooked: <p>I wasn’t aware of the issue with <code>BCEWithLogitsLoss</code> but <code>torch.round</code> definitely causes your gradients to equal 0 as well. So, make sure to remove <code>torch.round</code> as well to ensure gradient flow to all parameters! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>