Model does not train: Same loss in every epoch

melste: <p>Hey everyone,</p>
<p>this is my second pytorch implementation so far, for my first implementation the same happend; the model does not learn anything and outputs the same loss and accuracy for every epoch and even for each batch with an epoch. My personal guess is that something with the way I feed the data to the model is not correctly implemented.</p>
<p>I try to follow the basic tutorial for implementing a pytorch model: <a href="https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optimizing Model Parameters — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>I read similar topics on thepytorch forum, e.g. <a href="https://discuss.pytorch.org/t/same-values-in-every-epoch-when-training/95450" class="inline-onebox">Same values in every epoch when training</a></p>
<p>I’m using <code>nn.BCEWithLogitsLoss()</code> and already tried to overfit the model on a training sample of size 600.<br>
It’s rather a simple model, so I’m a bit confused that it’s not working at all.<br>
Find the code below:</p>
<p><strong>Load the data</strong></p>
<pre><code class="lang-auto">df = pd.read_csv(Path()/ "cleaned_data.csv", nrows=1000)
#delete id column
del df["Unnamed: 0"]
train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))])

X_train = train.iloc[:,1:]
Y_train = train.iloc[:,1]

X_validate = validate.iloc[:,1:]
Y_validate = validate.iloc[:,1]

X_test = validate.iloc[:,1:]
Y_test = validate.iloc[:,1]

x_train_pt = torch.from_numpy(X_train.values)
y_train_pt = torch.from_numpy(Y_train.values.reshape((-1,1)))

x_val_pt = torch.from_numpy(X_validate.values)
y_val_pt = torch.from_numpy(Y_validate.values.reshape((-1,1)))

x_test_pt = torch.from_numpy(X_test.values)
y_test_pt = torch.from_numpy(Y_test.values.reshape((-1,1)))
print(x_train_pt.shape) #torch.Size([600, 22])
print(y_train_pt.shape) #torch.Size([600, 1])

batch_size=100
train_dataset = TensorDataset(x_train_pt,y_train_pt) # create your datset
trainloader = DataLoader(train_dataset, batch_size=batch_size) # create your dataloader

val_dataset = TensorDataset(x_val_pt,y_val_pt)
valloader = DataLoader(val_dataset, batch_size=batch_size)

test_dataset = TensorDataset(x_test_pt,y_test_pt)
testloader = DataLoader(test_dataset, batch_size=batch_size)
</code></pre>
<p><strong>The Model</strong></p>
<pre><code class="lang-auto">class LendingClub(nn.Module):
    def __init__(self):
        super(LendingClub, self).__init__()
        self.linear = nn.Sequential(
            nn.Linear(22, 100),
            nn.ReLU(),
            nn.Linear(100, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, 8),
            nn.ReLU(),
            nn.Linear(8, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        return self.linear(x)
    
model = LendingClub()

def weights_init(m):
  if isinstance(m, nn.Linear):
      nn.init.uniform_(m.weight.data, -1,1)
      nn.init.zeros_(m.bias.data)

model.apply(weights_init)
</code></pre>
<p><strong>Training</strong></p>
<pre><code class="lang-auto">loss_fn = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X.float())
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        #if batch % 100 == 0:
        loss, current = loss.item(), batch * len(X)
        print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")


def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    test_loss, correct = 0, 0

    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X.float())
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()

    test_loss /= size
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")
</code></pre>
<pre><code class="lang-auto">model.train()
start = time.time()
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train_loop(trainloader, model, loss_fn, optimizer)
    test_loop(valloader, model, loss_fn)
end = time.time()
print(f"total training time in minutes: {(end-start)/60}")
</code></pre>
<p><strong>Output</strong></p>
<pre><code class="lang-auto">Epoch 1
-------------------------------
loss: 0.693147  [    0/  600]
loss: 0.693147  [  100/  600]
loss: 0.693147  [  200/  600]
loss: 0.693147  [  300/  600]
loss: 0.693147  [  400/  600]
loss: 0.693147  [  500/  600]
Epoch 2
-------------------------------
loss: 0.693147  [    0/  600]
loss: 0.693147  [  100/  600]
loss: 0.693147  [  200/  600]
loss: 0.693147  [  300/  600]
loss: 0.693147  [  400/  600]
loss: 0.693147  [  500/  600]
Epoch 3
-------------------------------
loss: 0.693147  [    0/  600]
loss: 0.693147  [  100/  600]
loss: 0.693147  [  200/  600]
loss: 0.693147  [  300/  600]
loss: 0.693147  [  400/  600]
loss: 0.693147  [  500/  600]
Epoch 4
-------------------------------
loss: 0.693147  [    0/  600]
loss: 0.693147  [  100/  600]
loss: 0.693147  [  200/  600]
loss: 0.693147  [  300/  600]
loss: 0.693147  [  400/  600]
loss: 0.693147  [  500/  600]
Epoch 5
-------------------------------
loss: 0.693147  [    0/  600]
loss: 0.693147  [  100/  600]
loss: 0.693147  [  200/  600]
loss: 0.693147  [  300/  600]
loss: 0.693147  [  400/  600]
loss: 0.693147  [  500/  600]
total training time in minutes: 0.011684314409891764
</code></pre>
<p>The model does not always output exactly the same loss, but the fact that for every epoch the losses are the same does not change.<br>
So the output from another run looked as following:</p>
<pre><code class="lang-auto">Epoch 1
-------------------------------
loss: 1.026536  [    0/  600]
loss: 0.980619  [  100/  600]
loss: 0.964926  [  200/  600]
loss: 1.120505  [  300/  600]
loss: 1.092735  [  400/  600]
loss: 0.910104  [  500/  600]
Test Error: 
 Accuracy: 0.0%, Avg loss: 0.010968 

Epoch 2
-------------------------------
loss: 1.026536  [    0/  600]
loss: 0.980619  [  100/  600]
loss: 0.964926  [  200/  600]
loss: 1.120505  [  300/  600]
loss: 1.092735  [  400/  600]
loss: 0.910104  [  500/  600]
Test Error: 
 Accuracy: 0.0%, Avg loss: 0.010968 

Epoch 3
-------------------------------
loss: 1.026536  [    0/  600]
loss: 0.980619  [  100/  600]
loss: 0.964926  [  200/  600]
loss: 1.120505  [  300/  600]
loss: 1.092735  [  400/  600]
loss: 0.910104  [  500/  600]
Test Error: 
 Accuracy: 0.0%, Avg loss: 0.010968
</code></pre>
<p>Thanks for your replies, any help is very much appreciated</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> expects logits as the model output, not probabilities, so you would have to remove the last <code>sigmoid</code> activation in your model. Besides that I don’t see any obvious issues in your code.</p>

melste: <p>Thanks for the fast reply.</p>
<p>If I deactivate the Sigmoid, the model indeed starts learning, though not really good. Besides that it starts outputting numbers outside of the interval [0,1].<br>
But my true labels are 0 or 1 in a Tensor of shape [batch_size, 1], so coming from Keras I chose Sigmoid as I want the output to be a probability between 0 and 1.</p>
<p>So which loss function would I have to choose for the model to work? Or should I remodel the true labels to [batch_size, 2], i.e. using one hot encodings, and then use simple BinaryCrossEntropy loss?</p>

ptrblck: <p>Logits are unbound and can contain any values in <code>[-Inf, Inf]</code>. <code>nn.BCEWithLogitsLoss</code> will apply the <code>sigmoid</code> internally in a numerically stable way, so the model outputs are expected to contain positive and negative values.<br>
You could use the (numerically less stable) <code>nn.BCELoss</code> with a <code>sigmoid</code> activation if you prefer it.</p>

pascal_notsawo: <aside class="quote no-group" data-username="melste" data-post="3" data-topic="121428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ecd19e/48.png" class="avatar"> melste:</div>
<blockquote>
<p>So which loss function would I have to choose for the model to work? Or should I remodel the true labels to [batch_size, 2], i.e. using one hot encodings, and then use simple BinaryCrossEntropy loss?</p>
</blockquote>
</aside>
<p>Add to what <a class="mention" href="/u/ptrblck">@ptrblck</a> says, keep a shape of [batch_size, 1]. And to know the output of your model, you can, once you have your logits (of shape [batch_size, 1] also), do :</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(0)
batch_size = 10
logits = torch.empty(batch_size, dtype=torch.float).uniform_(-10, 10)

pred = (logits &gt;= 0.).int()
# Or if you prefer to use the sigmoid (x &gt; 0 &lt;====&gt; 1 / (1+e-x) &gt; 0.5 )
pred = torch.sigmoid(logits).round().int() # = (torch.sigmoid(logits) &gt;= 0.5).int()
</code></pre>
<p>If you switch to a shape of [batch_size, 2] and use nn.BCEWithLogitsLoss, it will be like doing multi-label classification.</p>

melste: <p>I tried both of your proposal, but with no success. Although the loss changes, the pattern still remains the same across all epochs. It doesn’t matter if I don’t use <code>nn.Sigmoid()</code> as the last activation function and use <code>nn.BCEWithLogitsLoss</code> or no sigmoid and <code>nn.BCELoss</code>. Though I did not really understand how to incorporate what <a class="mention" href="/u/pascal_notsawo">@pascal_notsawo</a> suggested. I tried:</p>
<pre><code class="lang-auto">pred = model(X)
pred = torch.sigmoid(pred).round()
loss = loss_fn(pred, y)
</code></pre>
<p>This is the training performance for Sigmoid activation with BCELoss, also the accuracy of 1500% indicates that something is really wrong:</p>
<pre><code class="lang-auto">Epoch 1
-------------------------------
loss: 11.000000  [    0/  600]
loss: 21.000000  [  100/  600]
loss: 25.000000  [  200/  600]
loss: 22.000000  [  300/  600]
loss: 19.000000  [  400/  600]
loss: 14.000000  [  500/  600]
Test Error: Accuracy: 1500.0%, Avg loss: 0.150000 
...
Epoch 5
-------------------------------
loss: 11.000000  [    0/  600]
loss: 21.000000  [  100/  600]
loss: 25.000000  [  200/  600]
loss: 22.000000  [  300/  600]
loss: 19.000000  [  400/  600]
loss: 14.000000  [  500/  600]
Test Error: 
 Accuracy: 1500.0%, Avg loss: 0.150000
</code></pre>
<p>When I monitor the output of the model during training, I see that it correctly outputs a Tensor of shape [batch_size, 1] containing 0 or 1. I don’t really understand this output, as I use the sigmoid activation I would expect a number between 0 and 1 as output, but not exclusively 0 or 1.<br>
I further observed that the output generated by the model for the last batch in epoch 1 is identical to the output generated for the same batch in the last epoch. (I collected the outputs and compared the tensors after training) This seems to indicate to me, that the model does not update its parameters.</p>
<p><strong>So my actual question is at the moment, what am I doing wrong that the model does not output values between 0 and 1, but 0 or 1?</strong> If it would, using nn.BCELoss should be the right way to go. The same model architecture with BCELoss works btw in Keras.</p>
<p>I’d be open to suggestion on how to change the architecture of the model to make it work; my data looks as following and the task is to predict the label of <code>loan_status (0 or 1)</code>. (The data is not completely cleaned and preprocessed yet)</p>
<p><code>df.head()</code></p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>loan_status</th>
<th>int_rate</th>
<th>installment</th>
<th>annual_inc</th>
<th>loan_amnt</th>
<th>dti</th>
<th>open_acc</th>
<th>pub_rec</th>
<th>revol_bal</th>
<th>revol_util</th>
<th>…</th>
<th>seniority</th>
<th>term</th>
<th>emp_length</th>
<th>home_ownership</th>
<th>verification_status</th>
<th>purpose</th>
<th>initial_list_status</th>
<th>application_type</th>
<th>address</th>
<th>issue_d</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>1</td>
<td>-0.491799</td>
<td>-0.408291</td>
<td>117000.0</td>
<td>-0.492243</td>
<td>1.194178</td>
<td>0.913445</td>
<td>-0.34997</td>
<td>1.340611</td>
<td>-0.491258</td>
<td>…</td>
<td>0.357143</td>
<td>-1</td>
<td>0</td>
<td>2</td>
<td>-1</td>
<td>10</td>
<td>1</td>
<td>1</td>
<td>22690</td>
<td>2015</td>
</tr>
<tr>
<td>1</td>
<td>1</td>
<td>-0.368816</td>
<td>-0.662750</td>
<td>65000.0</td>
<td>-0.731551</td>
<td>0.646367</td>
<td>1.108239</td>
<td>-0.34997</td>
<td>-0.174517</td>
<td>-0.019706</td>
<td>…</td>
<td>0.157143</td>
<td>-1</td>
<td>1</td>
<td>0</td>
<td>-1</td>
<td>2</td>
<td>-1</td>
<td>1</td>
<td>5113</td>
<td>2015</td>
</tr>
<tr>
<td>2</td>
<td>1</td>
<td>-0.704225</td>
<td>0.299609</td>
<td>43057.0</td>
<td>0.177819</td>
<td>-0.564308</td>
<td>0.329061</td>
<td>-0.34997</td>
<td>-0.813871</td>
<td>1.575371</td>
<td>…</td>
<td>0.114286</td>
<td>-1</td>
<td>1</td>
<td>2</td>
<td>1</td>
<td>1</td>
<td>-1</td>
<td>1</td>
<td>5113</td>
<td>2015</td>
</tr>
<tr>
<td>3</td>
<td>1</td>
<td>-1.598649</td>
<td>-0.842348</td>
<td>54000.0</td>
<td>-0.827274</td>
<td>-1.896573</td>
<td>-1.034500</td>
<td>-0.34997</td>
<td>-0.495024</td>
<td>-1.323650</td>
<td>…</td>
<td>0.114286</td>
<td>-1</td>
<td>1</td>
<td>2</td>
<td>-1</td>
<td>1</td>
<td>-1</td>
<td>1</td>
<td>813</td>
<td>2014</td>
</tr>
<tr>
<td>4</td>
<td>0</td>
<td>0.811824</td>
<td>0.707861</td>
<td>55000.0</td>
<td>1.227783</td>
<td>1.685770</td>
<td>0.329061</td>
<td>-0.34997</td>
<td>-0.465887</td>
<td>0.656869</td>
<td>…</td>
<td>0.200000</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>-1</td>
<td>1</td>
<td>11650</td>
<td>2013</td>
</tr>
</tbody>
</table>
</div>

pascal_notsawo: <p><a class="mention" href="/u/melste">@melste</a>  what I gave as code was to be able to calculate things like accuracy, precision and other … Otherwise for the calculation of the loss, it remains :</p>
<ol>
<li>either you remove the Sigmoid layer at the output of the model : advised, because more stable</li>
</ol>
<pre><code class="lang-auto">loss_fn = nn.BCEWithLogitsLoss()

logits = model(...your inputs...)

loss = loss_fn(logits, y)
</code></pre>
<p>Then you can apply the sigmoid to the logits output as I mentioned in my post above, to know if your model has predicted 0 or 1.</p>
<pre><code class="lang-auto">prediction = (logits &gt;= 0.).int()
# Or if you prefer to use the sigmoid (x &gt; 0 &lt;====&gt; 1 / (1+e-x) &gt; 0.5 )
probability  = torch.sigmoid(logits)
prediction = probability.round().int() # = (probability &gt;= 0.5).int()
</code></pre>
<ol start="2">
<li>or you keep this sigmoid layer</li>
</ol>
<pre><code class="lang-auto">loss_fn = nn.BCELoss()

probability = model(...your inputs...)

loss = loss_fn(probability, y)
</code></pre>
<p>In this case your model returns directly a probability (between [0,1]), that you can also compare to 0.5 to know if your model has predicted 0 or 1.</p>
<pre><code class="lang-auto">prediction = probability.round().int() # = (probability &gt;= 0.5).int()
</code></pre>

melste: <p>I think I understood your answer now and why using <code>nn.BCEWithLogitsLoss()</code> together with Sigmoid is a bad idea. Thanks a lot!</p>
<p>So I’d like to do the second proposal. But what I don’t understand is, why the model outputs 0 or 1 and not values between 0 and 1.<br>
Output of the model (i.e. <code>pred</code>) looks like this:</p>
<pre><code class="lang-auto">tensor([[1.],
        [0.],
        [0.],
        [0.],
        ...
        [0.],
        [1.],
        [0.],
        [0.],
        [1.]], grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>
<p>I would like to have an output as in keras:</p>
<pre><code class="lang-auto">[4.5889354e-01],
       [1.0000000e+00],
       [9.9999785e-01],
       ...
       [8.9359963e-01],
       [1.0000000e+00],
       [2.6206519e-05],
       [1.0000000e+00],
       [1.0000000e+00]], dtype=float32)&gt;
</code></pre>

pascal_notsawo: <p>It seems you are using <code>.round()</code> in the wrong place.<br>
As mentioned in each of my comments, knowing that the probability (after the sigmoid) is in the range [0,1], <code>probability.round()</code> is just an elegant way of doing <code>probability &gt;= 0.5</code>, which returns 0. (False) if the value is in the range [0, 0.5[ and 1. (True) if the value is in the interval [0.5, 1]<br>
And in both cases, I do <code>.int()</code> to convert them to integers (0. becomes 0 and 1. becomes 1)<br>
So don’t use it anyhow.</p>
<p>It is important to distinguish between logits, probabilities and the final prediction of the model.</p>
<p>BCE with logits waits for the logits, BCE simply waits for the probabilities,   acc_score … waits for the predictions</p>

melste: <p>I don’t use <code>.round()</code> anywhere. I’m using what you suggested for nn.BCELoss().</p>
<aside class="quote no-group" data-username="pascal_notsawo" data-post="7" data-topic="121428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pascal_notsawo/48/22829_2.png" class="avatar"> pascal_notsawo:</div>
<blockquote>
<pre><code class="lang-auto">loss_fn = nn.BCELoss()

probability = model(...your inputs...)

loss = loss_fn(probability, y)
</code></pre>
</blockquote>
</aside>
<p>So training loop is simple:<br>
The output I reported in the post before was <code>pred</code> in the training loop.</p>
<pre><code class="lang-auto">for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        #if batch % 100 == 0:
        loss, current = loss.item(), batch * len(X)
        print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>The model is the same as described in my first post.</p>
<pre><code class="lang-auto">class LendingClub(nn.Module):
    def __init__(self):
        super(LendingClub, self).__init__()
        self.linear = nn.Sequential(
            nn.Linear(22, 100),
            nn.ReLU(),
            nn.Linear(100, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, 8),
            nn.ReLU(),
            nn.Linear(8, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        return self.linear(x)
    
model = LendingClub()
</code></pre>

pascal_notsawo: <aside class="quote no-group" data-username="melste" data-post="8" data-topic="121428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ecd19e/48.png" class="avatar"> melste:</div>
<blockquote>
<pre><code class="lang-auto">tensor([[1.],
        [0.],
        [0.],
        [0.],
        ...
        [0.],
        [1.],
        [0.],
        [0.],
        [1.]], grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>
</blockquote>
</aside>
<p>This is the value of which variable in your pipeline?</p>

melste: <p>It’s <code>pred</code> in the training loop. So far I have seen tensor with only 0, only 1, 0 or 1, but never a value in between…<br>
I collected it this way:</p>
<pre><code class="lang-auto">predictions = []
...
for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X)
        predictions.append(pred)
        loss = loss_fn(pred, y)
        ....
</code></pre>

melste: <p>Does anyone have another suggestion?<br>
I also tried to run the same model on GoogleColab to check whether something on my device is wrong, but the performance is the same on GoogleColab.</p>
<p>So far I could narrow the problem down to the fact that the predicted output of the model is identical for each batch out of each epoch.<br>
The predictions of the model is always a tensor with exclusively zeros and/or ones.<br>
The same model (identical architecture) with exactly the same data performs ok when I use Keras. So the data does not seem to be the problem.<br>
If I deactivate the Sigmoid activation of the last layer and use BCEWithLogitsLoss, the model does not get stuck.</p>

pascal_notsawo: <p>Can you share the link to your colab notebook?</p>

melste: <p>Hey, sure. Here comes the link: <a href="https://colab.research.google.com/drive/1RDfoeOpZ1kzL9qEBEreul9nACFOhwPr6?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a></p>
<p>Though I don’t really know how to make the data accessible for you…</p>

melste: <p>Could you find an error in the pytorch implementation?</p>

ptrblck: <p>Your model is generally able to overfit a small dataset as seen here:</p>
<pre><code class="lang-python">class LendingClub(nn.Module):
    def __init__(self):
        super(LendingClub, self).__init__()
        self.linear = nn.Sequential(
            nn.Linear(22, 100),
            nn.ReLU(),
            nn.Linear(100, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, 8),
            nn.ReLU(),
            nn.Linear(8, 1),
        )

    def forward(self, x):
        return self.linear(x)
    
model = LendingClub()

def weights_init(m):
  if isinstance(m, nn.Linear):
      nn.init.uniform_(m.weight.data, -1,1)
      nn.init.zeros_(m.bias.data)
model.apply(weights_init)

loss_fn = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

data = torch.randn(8, 22)
target = torch.randint(0, 2, (8, 1)).float()

for epoch in range(100):
    pred = model(data)
    loss = loss_fn(pred, target)
        
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print('epoch {}, loss {}, acc {}'.format(
        epoch, loss.item(), ((pred&gt;0.0)==target).float().mean()))
</code></pre>
<p>if you use <code>nn.BCEWithLogitsLoss</code> (in your current code snippet it seems you’ve experimented with different approaches such as using <code>nn.BCELoss</code> + <code>sigmoid(output)</code>), so you could also try to overfit a small dataset using the real data by playing around with some hyperparameters.</p>