Why does my deep-learning model get only 30% cater rate even after 1000 epochs

Small_Red_69: <p>Hi, friends,</p>
<p>The user case of my self-created PyTorch Deep Learning model is based on patients’ medical-appoint booking behaviours. In details, it is to predict how many days in advance (except weekends and public holidays) a patient will book a medical appointment. The range is 2 to 20 days.</p>
<p>Actually this is quite a simple user case. However, the cater rate of my model is all the way very low: even after 1,000 epochs, the accuracy rate is only 30.9%, and the average loss is still as high as 2.058354.</p>
<p>In my model, the data tensor is indeed a vector composed of the following 7 fields:</p>
<ol>
<li>Gender: 1 – male, 2 – female</li>
<li>Age</li>
<li>Area: I take the first 3 digits of a patient’s residential postal codes and map to an integer from 0 to 999.</li>
<li>Medical Examination: 1 – yes, 0 – no</li>
<li>Blood Test: 1 – yes, 0 – no</li>
<li>Urine Test: 1 – yes, 0 – no</li>
<li>Fasting: 1 – yes, 0 – no<br>
The labels of my model are taken by subtracting the booking-in-advance days by 2, so I get the range from 0 to 18, totally 19 categories.<br>
Totally I prepare 377 training data and 94 testing data for the deep-learning model.</li>
</ol>
<p>Basically my deep-learning model just follows the Fashion-MNIST learning model from the tutorials in <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> official website (see the hyperlink <a href="https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optimizing Model Parameters — PyTorch Tutorials 1.12.0+cu102 documentation</a>). In the model, I also take the following parameters: the size of middle layer in the Neural Network is 512, the learning rate is 0.01 and the batch size is 10. (At the end of this post, I will attach the full set of my python source codes for your reference).</p>
<p>Can any experts help me to analyse and diagnose why my cater rate is so low? I feel that the low accuracy rate may be caused by one or multiple reasons as below:</p>
<ol>
<li>Training data size is too small.</li>
<li>Running epochs (so far 1000 epochs) are not enough.</li>
<li>The learning model is not efficient enough (do I need to consider some other algorithms the model? e.g., is the loss function good enough)?</li>
<li>Other parameters, e.g., batch size, learning rate, and so on, may be misconfigured.</li>
</ol>
<p>Thanks a lot for any help. If anybody needs to get more info to investigate the issue, please kindly let me know, and I will try my best to provide.</p>
<h1>
<a name="my-source-codes-of-the-pytorch-deep-learning-model-1" class="anchor" href="#my-source-codes-of-the-pytorch-deep-learning-model-1"></a>My source codes of the Pytorch deep-learning model:</h1>
<p>import torch<br>
from torch import nn<br>
from torch.utils.data import Dataset, DataLoader<br>
from torchvision import datasets<br>
from torchvision.transforms import ToTensor, Lambda<br>
import numpy as np<br>
import pandas as pd</p>
<p>class CustomDataset(Dataset):<br>
‘Characterizes a dataset for PyTorch’<br>
def <strong>init</strong>(self, csv_file):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file.<br>
“”"<br>
raw_data = pd.read_csv(csv_file)<br>
raw_data = torch.tensor(raw_data.to_numpy())<br>
x_size = list(raw_data.size())[1]<br>
self.data_tensor = raw_data[:,:x_size-1].clone()<br>
self.data_tensor = self.data_tensor.type(torch.float32)<br>
self.label_tensor = raw_data[:,x_size-1:].clone()<br>
self.label_tensor = self.label_tensor.flatten()<br>
print(f"self.label_tensor.size() = {self.label_tensor.size()}")<br>
def <strong>len</strong>(self):<br>
‘Denotes the total number of samples’<br>
return len(self.data_tensor)<br>
def <strong>getitem</strong>(self, index):<br>
data, label = self.data_tensor[index], self.label_tensor[index]<br>
return data, label</p>
<p>training_data_csv_file = “D:\Tools\PyTorch\Deep-Learning-Model\input_data\training_data.csv”<br>
training_data = CustomDataset(training_data_csv_file)<br>
test_data_csv_file = “D:\Tools\PyTorch\Deep-Learning-Model\input_data\testing_data.csv”<br>
test_data = CustomDataset(test_data_csv_file)</p>
<p>train_dataloader = DataLoader(training_data, batch_size=10)<br>
test_dataloader = DataLoader(test_data, batch_size=10)</p>
<p>class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 19),<br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>model = NeuralNetwork()</p>
<p>def train_loop(dataloader, model, loss_fn, optimizer):<br>
size = len(dataloader.dataset)<br>
print(f’size = {size}‘);<br>
for batch, (X, y) in enumerate(dataloader):<br>
# Compute prediction and loss<br>
pred = model(X)<br>
print(f’pred = {pred}’);<br>
print(f’y = {y}‘);<br>
loss = loss_fn(pred, y)<br>
print(f’loss = {loss}’);<br>
# Backpropagation<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>    if batch % 100 == 0:
        loss, current = loss.item(), batch * len(X)
        print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>learning_rate = 0.01<br>
batch_size = 10</p>
<p>def test_loop(dataloader, model, loss_fn):<br>
size = len(dataloader.dataset)<br>
num_batches = len(dataloader)<br>
test_loss, correct = 0, 0</p>
<pre><code>with torch.no_grad():
    for X, y in dataloader:
        pred = model(X)
        test_loss += loss_fn(pred, y).item()
        correct += (pred.argmax(1) == y).type(torch.float).sum().item()

test_loss /= num_batches
correct /= size
print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")
</code></pre>
<p>loss_fn = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<h1>
<a name="epochs-1000-for-t-in-rangeepochs-printfepoch-t1n-train_looptrain_dataloader-model-loss_fn-optimizer-test_looptest_dataloader-model-loss_fn-printdone-2" class="anchor" href="#epochs-1000-for-t-in-rangeepochs-printfepoch-t1n-train_looptrain_dataloader-model-loss_fn-optimizer-test_looptest_dataloader-model-loss_fn-printdone-2"></a>epochs = 1000<br>
for t in range(epochs):<br>
print(f"Epoch {t+1}\n-------------------------------")<br>
train_loop(train_dataloader, model, loss_fn, optimizer)<br>
test_loop(test_dataloader, model, loss_fn)<br>
print(“Done!”)</h1>

KFrank: <p>Hi Small!</p>
<aside class="quote no-group quote-modified" data-username="Small_Red_69" data-post="1" data-topic="158194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/5f9b8f/48.png" class="avatar"> Small_Red_69:</div>
<blockquote>
<p>it is to predict how many days in advance (except weekends and public holidays) a patient will book a medical appointment. The range is 2 to 20 days.<br>
…<br>
loss_fn = nn.CrossEntropyLoss()</p>
</blockquote>
</aside>
<p>My speculation is that you are treating your problem as a <em>classification</em><br>
problem, that is, you are trying to predict the number of days exactly.</p>
<p>I think you will get better results treating your use case as a <em>regression.</em></p>
<p>A bit of explanation:</p>
<p>Let’s say that the actual number of days you are trying to predict is 14.<br>
When training as classification, if your model predicts 14, it’s right, and<br>
you get a low loss function.  But if your model predicts 13 or 15, it’s<br>
wrong, and it’s <em>just as wrong</em> as if it had predicted 3 or 22, even though<br>
a prediction of 13 or 15 is really quite good, even if not perfect.</p>
<p>In a regression, you will penalize your model more for being way off (e.g.,<br>
3 or 22) than for being close, but not perfect (e.g., 13 or 15).</p>
<p>Regression seems to me to be a better conceptual fit for what you are<br>
actually trying to accomplish.</p>
<p>Furthermore, even though your use case is straightforward and easy to<br>
describe, I could see it being quite a difficult problem if your measure of<br>
success is getting your predictions <em>exactly</em> right,.  Realistically, based on<br>
your input data, how is your model supposed to predict 14 instead of 15<br>
(for the example where the correct result is 14)?  That seems hard, so an<br>
accuracy rate of 30% doesn’t sound so bad.</p>
<p>To perform a regression, change your final <code>Linear</code> layer to have<br>
<code>out_features = 1</code>, that is, <code>Linear (512, 1)</code>, and use <code>MSELoss</code> as<br>
your loss criterion.  (Let your predictions be floating-point numbers, e.g.,<br>
“predicted number of days = 14.72.”)</p>
<p>Then measure your performance based not on how often your model<br>
gets the number of days exactly right, but rather, on how close it usually<br>
is.  So use the mean-squared-error or root-mean-squared-error of your<br>
predictions to judge how well your model is doing.</p>
<p>Best.</p>
<p>K. Frank</p>

Small_Red_69: <p>Great!<br>
Thanks Frank, I will make a retry.</p>

Small_Red_69: <p>Hi, Frank,</p>
<p>Right now I have customized my Pytorch source codes to adopt the Regression solution instead of the Classification solution.</p>
<p>The relevant customization of codes is highlighted below:</p>
<ol>
<li>
<p>In Layer 2, change the “out_features” to be 1:<br>
====================================================================<br>
class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 1),<br>
)<br>
…<br>
====================================================================</p>
</li>
<li>
<p>Change the data type of both data and label tensors from Long to float32:<br>
====================================================================<br>
class CustomDataset(Dataset):<br>
‘Characterizes a dataset for PyTorch’<br>
def <strong>init</strong>(self, csv_file):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file.<br>
“”"<br>
raw_data = pd.read_csv(csv_file)<br>
raw_data = torch.tensor(raw_data.to_numpy())<br>
x_size = list(raw_data.size())[1]<br>
self.data_tensor = raw_data[:,:x_size-1].clone()<br>
self.data_tensor = self.data_tensor.type(torch.float32)<br>
self.label_tensor = raw_data[:,x_size-1:].clone()<br>
self.label_tensor = self.label_tensor.type(torch.float32)<br>
self.label_tensor = self.label_tensor.flatten()<br>
…<br>
====================================================================</p>
</li>
</ol>
<p>However, when I was running the new model, the values of the predicted tensor are full of “nan” (see the output text shown below):</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
y = tensor([ 5.,  4.,  3.,  6.,  3., 10.,  8.,  5.,  3.,  1.])<br>
loss = nan<br>
pred = tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan]], grad_fn=)<br>
y = tensor([8., 5., 1., 0., 4., 6., 1., 6., 5., 8.])<br>
loss = nan<br>
pred = tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan],<br>
[nan]], grad_fn=)<br>
y = tensor([8., 1., 5., 5., 4., 1., 2.])<br>
loss = nan<br>
Test Error:<br>
Accuracy: 6.4%, Avg loss:      nan<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>I suspect that the function in layer 2 is totally wrong: in a regression deep-learning model, we should use a sigmoid function instead of a linear function, right?</p>
<p>Thanks.</p>

Small_Red_69: <p>Hi, Frank,</p>
<h1>
<a name="just-now-i-changed-the-final-layer-of-nn-to-be-a-sigmoid-function-instead-of-a-linear-function-see-the-code-snippet-below-1" class="anchor" href="#just-now-i-changed-the-final-layer-of-nn-to-be-a-sigmoid-function-instead-of-a-linear-function-see-the-code-snippet-below-1"></a>Just now I changed the final layer of NN to be a sigmoid function instead of a linear function (see the code snippet below):</h1>
<h1>
<a name="class-neuralnetworknnmodule-def-__init__self-superneuralnetwork-self__init__-selfflatten-nnflatten-selflinear_relu_stack-nnsequential-nnlinear7-512-nnrelu-nnlinear512-1-nnrelu-nnsigmoid-2" class="anchor" href="#class-neuralnetworknnmodule-def-__init__self-superneuralnetwork-self__init__-selfflatten-nnflatten-selflinear_relu_stack-nnsequential-nnlinear7-512-nnrelu-nnlinear512-1-nnrelu-nnsigmoid-2"></a>class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 1),<br>
nn.ReLU(),<br>
nn.Sigmoid(),<br>
)</h1>
<p>Then I run the model, but this time it returns all 1 values in the predicted tensor (see the output text below):<br>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
y = tensor([ 5.,  4.,  3.,  6.,  3., 10.,  8.,  5.,  3.,  1.])<br>
loss = 20.799999237060547<br>
pred = tensor([[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.]], grad_fn=)<br>
y = tensor([8., 5., 1., 0., 4., 6., 1., 6., 5., 8.])<br>
loss = 19.0<br>
pred = tensor([[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.],<br>
[1.]], grad_fn=)<br>
y = tensor([8., 1., 5., 5., 4., 1., 2.])<br>
loss = 13.0<br>
Test Error:<br>
Accuracy: 6.4%, Avg loss: 7.595000</p>
<p>Done!<br>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>Any idea on this issue?</p>
<p>Thanks for your further help.</p>

KFrank: <p>Hi Small!</p>
<aside class="quote no-group quote-modified" data-username="Small_Red_69" data-post="5" data-topic="158194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/5f9b8f/48.png" class="avatar"> Small_Red_69:</div>
<blockquote>
<p>Just now I changed the final layer of NN to be a sigmoid function instead of a linear function<br>
…<br>
Then I run the model, but this time it returns all 1 values in the predicted tensor</p>
</blockquote>
</aside>
<p>No, replacing you final <code>Linear</code> layer with a <code>Sigmoid</code> “activation” doesn’t<br>
make sense.  <code>Sigmoid</code> returns a value that ranges from <code>0.0</code> to <code>1.0</code>.  You<br>
are trying to predict a number of days that ranges from <code>0</code> to <code>19</code> so it’s<br>
only natural that your network will learn to always predict <code>1.0</code> as that’s<br>
(almost) always the value that it can predict that is closest to the target<br>
value.</p>
<p>Going back to your previous architecture (with the three <code>Linear</code> layers),<br>
I don’t see anything that is obviously wrong.  There is nothing wrong with<br>
using <code>ReLU</code> for your activations.  (Once you get things working, you could<br>
also try <code>Sigmoid</code> activations and see which works better for your use case.)</p>
<p>To debug the <code>nan</code>s you are getting, first read through your code looking<br>
for any obvious bugs.  (Always a good first step.)</p>
<p>Do one forward pass.  Do you get a sensible (and non-<code>nan</code>) loss?  Then<br>
do one backward pass.  Do you get non-<code>nan</code> gradients?</p>
<p>Then try optimizing with plain-vanilla <code>SGD</code> (no momentum nor weight<br>
decay) with a very small learning rate.  Is everything okay after a single<br>
batch / update?  Try training for a few batches.  Is your loss tending to<br>
go down (even if it jitters up a little sometimes)?</p>
<p>If you can get that working, try increasing your learning rate so that you<br>
can train faster, but without having your training become unstable.</p>
<p>If you keep getting <code>nan</code>s with these simple tests, try using autograd’s<br>
<a href="https://pytorch.org/docs/1.12/autograd.html#anomaly-detection" rel="noopener nofollow ugc">anomaly detection</a> to see if that helps you track the issue down.</p>
<p>If things are under control, you can try turning on momentum, again in<br>
order to train faster.  (You could also try the <code>Adam</code> optimizer.  It often trains<br>
much faster, but can jump around more and be less stable.)</p>
<p>If you get to a stage where you can train stably for a while, but then your<br>
training becomes unstable and / or you get <code>nan</code>s, you could try turning<br>
on weight decay to see if that helps stabilize things.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Small_Red_69: <p>Hi, Frank,</p>
<p>Thanks for your feedback.<br>
Btw, do you feel that the size of the middle-layer (512) is too big? Can I reduce them to about 20-30? How can we decide the size of the middle-layer based on the size of in/out features?</p>

Small_Red_69: <p>Hi, Frank,</p>
<p>Just now I tried to print the weights of all layers of my deep-learning model, but surprisingly found that except for Layer 0, both Layer 2 &amp; 4 get all weight values as “nan” (see the output text below)!</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
Model Parameters:</p>
<p>Layer: linear_relu_stack.0.weight | Size: torch.Size([20, 7]) | Values : tensor([[ 0.0312,  0.1015, -0.2662,  0.3075,  0.2584, -0.0975, -0.0711],<br>
[-0.0832, -0.2944, -0.2128, -0.2432,  0.0232,  0.3756, -0.1670]],<br>
grad_fn=)</p>
<p>Layer: linear_relu_stack.0.bias | Size: torch.Size([20]) | Values : tensor([-0.0503, -0.1474], grad_fn=)</p>
<p>Layer: linear_relu_stack.2.weight | Size: torch.Size([20, 20]) | Values : tensor([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],<br>
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]],<br>
grad_fn=)</p>
<p>Layer: linear_relu_stack.2.bias | Size: torch.Size([20]) | Values : tensor([nan, nan], grad_fn=)</p>
<p>Layer: linear_relu_stack.4.weight | Size: torch.Size([1, 20]) | Values : tensor([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]],<br>
grad_fn=)</p>
<p>Layer: linear_relu_stack.4.bias | Size: torch.Size([1]) | Values : tensor([nan], grad_fn=)<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>But how can be like this?</p>
<p>For your reference, I share my code snippet to build up the NN model below. Actually what I have done is nothing more than changing the loss function from nn.CrossEntropyLoss() to nn.MSELoss(), but somehow hit such an issue.</p>
<p>======================================================================<br>
import torch<br>
from torch import nn<br>
from torch.utils.data import Dataset, DataLoader<br>
from torchvision import datasets<br>
from torchvision.transforms import ToTensor, Lambda<br>
import numpy as np<br>
import pandas as pd</p>
<p>class CustomDataset(Dataset):<br>
‘Characterizes a dataset for PyTorch’<br>
def <strong>init</strong>(self, csv_file):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file.<br>
“”"<br>
raw_data = pd.read_csv(csv_file)<br>
raw_data = torch.tensor(raw_data.to_numpy())<br>
x_size = list(raw_data.size())[1]<br>
self.data_tensor = raw_data[:,:x_size-1].clone()<br>
self.data_tensor = self.data_tensor.type(torch.float32)<br>
self.label_tensor = raw_data[:,x_size-1:].clone()<br>
self.label_tensor = self.label_tensor.type(torch.float32)<br>
self.label_tensor = self.label_tensor.flatten()<br>
print(f"self.label_tensor.size() = {self.label_tensor.size()}")<br>
def <strong>len</strong>(self):<br>
‘Denotes the total number of samples’<br>
return len(self.data_tensor)<br>
def <strong>getitem</strong>(self, index):<br>
data, label = self.data_tensor[index], self.label_tensor[index]<br>
return data, label</p>
<p>training_data_csv_file = “D:\Tools\PyTorch\Medex-Deep-Learning-Model\input_data\training_data.csv”<br>
training_data = CustomDataset(training_data_csv_file)<br>
test_data_csv_file = “D:\Tools\PyTorch\Medex-Deep-Learning-Model\input_data\testing_data.csv”<br>
test_data = CustomDataset(test_data_csv_file)</p>
<p>train_dataloader = DataLoader(training_data, batch_size=10)<br>
test_dataloader = DataLoader(test_data, batch_size=10)</p>
<p>class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 20),<br>
nn.ReLU(),<br>
nn.Linear(20, 20),<br>
nn.ReLU(),<br>
nn.Linear(20, 1),<br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>model = NeuralNetwork()</p>
<p>def train_loop(dataloader, model, loss_fn, optimizer):<br>
size = len(dataloader.dataset)<br>
print(f’size = {size}‘);<br>
for batch, (X, y) in enumerate(dataloader):<br>
# Compute prediction and loss<br>
pred = model(X)<br>
print(f’pred = {pred}’);<br>
print(f’y = {y}‘);<br>
loss = loss_fn(pred, y)<br>
# print(f’loss = {loss}’);<br>
# Backpropagation<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>    if batch % 100 == 0:
        loss, current = loss.item(), batch * len(X)
        print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>learning_rate = 0.01<br>
batch_size = 10</p>
<p>def test_loop(dataloader, model, loss_fn):<br>
size = len(dataloader.dataset)<br>
num_batches = len(dataloader)<br>
test_loss, correct = 0, 0</p>
<pre><code>with torch.no_grad():
    for X, y in dataloader:
        pred = model(X)
        print(f'testing--&gt;pred = {pred}');
        print(f'testing--&gt;y = {y}');
        loss = loss_fn(pred, y).item()
        print(f'testing--&gt;loss = {loss}');
        test_loss += loss
        print(f'testing--&gt;pred.argmax(1) = {pred.argmax(1)}');
        print(f'testing--&gt;(pred.argmax(1) == y) = {(pred.argmax(1) == y)}');
        corr = (pred.argmax(1) == y).type(torch.float).sum().item()
        print(f'testing--&gt;corr = {corr}');
        correct += corr
print(f'testing--&gt;test_loss = {test_loss}, num_batches = {num_batches}');
test_loss /= num_batches
print(f'testing--&gt;correct = {correct}, size = {size}');
correct /= size
print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")
</code></pre>
<p>loss_fn = nn.MSELoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<p>epochs = 10<br>
for t in range(epochs):<br>
print(f"Epoch {t+1}\n-------------------------------")<br>
train_loop(train_dataloader, model, loss_fn, optimizer)<br>
test_loop(test_dataloader, model, loss_fn)<br>
print(“Done!”)</p>
<p>print(f"Model structure: {model}\n\n")</p>
<h1>
<a name="printfmodel-parametersnn-for-name-param-in-modelnamed_parameters-printflayer-name-size-paramsize-values-param2-n-1" class="anchor" href="#printfmodel-parametersnn-for-name-param-in-modelnamed_parameters-printflayer-name-size-paramsize-values-param2-n-1"></a>print(f"Model Parameters:\n\n")<br>
for name, param in model.named_parameters():<br>
print(f"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \n")</h1>
<p>Thanks again for your kind help.</p>

Small_Red_69: <p>Hi, Frank,</p>
<p>I refer to your post on 08 Aug.</p>
<p>" Do one forward pass. Do you get a sensible (and non-<code>nan</code> ) loss?"</p>
<p>just now I was testing the learning model with only 1 epoch, but it immediately hit the “nan” issue after running the loss function nn.MSELoss(). Please refer to the output text below:</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
testing–&gt;y = tensor([0., 4., 0., 1., 2., 1., 1., 2., 2., 1.])<br>
testing–&gt;loss = nan<br>
testing–&gt;pred.argmax(1) = tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])<br>
testing–&gt;(pred.argmax(1) == y) = tensor([ True, False,  True, False, False, False, False, False, False, False])<br>
testing–&gt;corr = 2.0<br>
testing–&gt;pred = tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]])<br>
testing–&gt;y = tensor([3., 1., 6., 3.])<br>
<em>D:\Tools\Python\lib\site-packages\torch\nn\modules\loss.py:530: UserWarning: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([4, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.</em><br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
<strong>testing–&gt;loss = nan</strong><br>
testing–&gt;pred.argmax(1) = tensor([0, 0, 0, 0])<br>
testing–&gt;(pred.argmax(1) == y) = tensor([False, False, False, False])<br>
testing–&gt;corr = 0.0<br>
testing–&gt;test_loss = nan, num_batches = 10<br>
testing–&gt;correct = 6.0, size = 94<br>
Test Error:<br>
Accuracy: 6.4%, Avg loss:      nan<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>Do you have any idea on this MSELoss() “nan” value issue?</p>
<p>Thanks.</p>

Small_Red_69: <p>Sorry, Frank,</p>
<p>Please ignore my above reply, because the info is incorrect.<br>
Just now I use “findstr” command to single out all training loss values, and get the real scenario: the loss value is somehow increasing rapidly and exponentially, and it quickly runs out of the range, and hit the “nan” issue. Please refer to the output text below:</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
D:\Tools\PyTorch\Medex-Deep-Learning-Model\Program&gt;python Medex_Booking_Model.py | findstr /c:"Training–&gt;loss = " | more<br>
D:\Tools\Python\lib\site-packages\torch\nn\modules\loss.py:530: UserWarning: Using a target size (torch.Size([10])) that is different to the input size (torch.Size([10, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
D:\Tools\Python\lib\site-packages\torch\nn\modules\loss.py:530: UserWarning: Using a target size (torch.Size([7])) that is different to the input size (torch.Size([7, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
D:\Tools\Python\lib\site-packages\torch\nn\modules\loss.py:530: UserWarning: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([4, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
<strong>Training–&gt;loss = 28.644052505493164</strong><br>
<strong>Training–&gt;loss = 9922584576.0</strong><br>
<strong>Training–&gt;loss = 2.5872629588033536e+16</strong><br>
<strong>Training–&gt;loss = 2.4454491212850324e+24</strong><br>
<strong>Training–&gt;loss = 9.77978754869175e+20</strong><br>
<strong>Training–&gt;loss = 9.392507535610442e+20</strong><br>
<strong>Training–&gt;loss = 9.020564390322656e+20</strong><br>
<strong>Training–&gt;loss = 8.663350126878696e+20</strong><br>
<strong>Training–&gt;loss = 8.320280684701889e+20</strong><br>
<strong>Training–&gt;loss = 7.990798039650906e+20</strong><br>
<strong>Training–&gt;loss = 7.674363167145344e+20</strong><br>
<strong>Training–&gt;loss = 7.370456745853174e+20</strong><br>
<strong>Training–&gt;loss = 7.078588305627477e+20</strong><br>
<strong>Training–&gt;loss = 6.798274413195753e+20</strong><br>
<strong>Training–&gt;loss = 6.529064004907822e+20</strong><br>
<strong>Training–&gt;loss = 6.270512350300482e+20</strong><br>
<strong>Training–&gt;loss = 6.022201459033317e+20</strong><br>
<strong>Training–&gt;loss = 5.783721785015214e+20</strong><br>
<strong>Training–&gt;loss = 5.5546866519969156e+20</strong><br>
<strong>Training–&gt;loss = 5.457337345065749e+34</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
<strong>Training–&gt;loss = nan</strong><br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>Pls help me to analyze why the MSE loss value can get such a terrible exponential increase.<br>
Thanks!</p>

ptrblck: <p>Don’t ignore this warning as it’s real:</p>
<pre><code class="lang-auto">UserWarning: Using a target size (torch.Size([10])) that is different to the input size (torch.Size([10, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<p>Fix the shape of the model output and the target and make sure they are matching. Otherwise the loss calculation will broadcast one tensor and will thus calculate a wrong loss value.</p>

Small_Red_69: <p>Now I come to know why the loss value keeps on increasing in such a terrible way: since there are 19 labels, with the range from 0 to 18, if we get an output value 1, then the MSE loss value will be quickly hiked to (18-1)^2=289. In next round, the value will further hike to (289-18)^2 = 271^2…, so on and so forth, this is really a horrible exponential “divergent” trend.</p>
<p>To prevent the divergency trend from happening, I plan to take the following two approaches: (1) I will still apply a sigmoid function in the final layer of the NN, to limit the range of any output value between 0 and 1; (2) The crucial point is that the label values are too big, must also be confined between 0 and 1. So I will do this way: I redefine the label 0 to be new label value 1/19, 1 to be 2/19, … so on and so forth, then the original label value 18 to be projected as new value 19/19, i.e., 1.</p>
<p>By this way, hopefully we can control the loss values between 0 and 1, and keep a convergent trend during the training process.</p>

Small_Red_69: <p>Hi, ptrblck,</p>
<p>Thank you for the reminding. I have fixed the issue by flattening the prev tendor for 2D to 1D (see the codes below):</p>
<h1>
<a name="pred-modelx-pred-predflatten-1" class="anchor" href="#pred-modelx-pred-predflatten-1"></a>===========================================================================<br>
…<br>
pred = model(X)<br>
<strong>pred = pred.flatten()</strong><br>
…</h1>

Small_Red_69: <p>Dear friends,</p>
<p>I have managed to implement the sigmoid function in the final layer of NN, and cater the label values to be within the range of [0,1]. (as mentioned in the last post). My new NN structure is shown by Python source codes as below:</p>
<p>===================================================================<br>
class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 20),<br>
nn.ReLU(),<br>
nn.Linear(20, 1),<br>
nn.ReLU(),<br>
<strong>nn.Sigmoid(),</strong><br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>===================================================================</p>
<p>Also I managed to cast the label data to be between 0 and 1. (see my Python source codes to initialize the label tensor below).</p>
<h1>
<a name="class-customdatasetdataset-characterizes-a-dataset-for-pytorch-def-__init__self-csv_file-args-csv_file-string-path-to-the-csv-file-raw_data-pdread_csvcsv_file-raw_data-torchtensorraw_datato_numpy-x_size-listraw_datasize1-selfdata_tensor-raw_datax_size-1clone-selfdata_tensor-selfdata_tensortypetorchfloat32-selflabel_tensor-raw_datax_size-1clone-selflabel_tensor-selflabel_tensortypetorchfloat32-selflabel_tensor-selflabel_tensorflatten-projected-to-0-181-218-1718-1-selflabel_tensor-selflabel_tensordiv180-printfselflabel_tensorsize-selflabel_tensorsize-def-__len__self-denotes-the-total-number-of-samples-return-lenselfdata_tensor-def-__getitem__self-index-data-label-selfdata_tensorindex-selflabel_tensorindex-return-data-label-1" class="anchor" href="#class-customdatasetdataset-characterizes-a-dataset-for-pytorch-def-__init__self-csv_file-args-csv_file-string-path-to-the-csv-file-raw_data-pdread_csvcsv_file-raw_data-torchtensorraw_datato_numpy-x_size-listraw_datasize1-selfdata_tensor-raw_datax_size-1clone-selfdata_tensor-selfdata_tensortypetorchfloat32-selflabel_tensor-raw_datax_size-1clone-selflabel_tensor-selflabel_tensortypetorchfloat32-selflabel_tensor-selflabel_tensorflatten-projected-to-0-181-218-1718-1-selflabel_tensor-selflabel_tensordiv180-printfselflabel_tensorsize-selflabel_tensorsize-def-__len__self-denotes-the-total-number-of-samples-return-lenselfdata_tensor-def-__getitem__self-index-data-label-selfdata_tensorindex-selflabel_tensorindex-return-data-label-1"></a>===================================================================<br>
class CustomDataset(Dataset):<br>
‘Characterizes a dataset for PyTorch’<br>
def <strong>init</strong>(self, csv_file):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file.<br>
“”"<br>
raw_data = pd.read_csv(csv_file)<br>
raw_data = torch.tensor(raw_data.to_numpy())<br>
x_size = list(raw_data.size())[1]<br>
self.data_tensor = raw_data[:,:x_size-1].clone()<br>
self.data_tensor = self.data_tensor.type(torch.float32)<br>
self.label_tensor = raw_data[:,x_size-1:].clone()<br>
self.label_tensor = self.label_tensor.type(torch.float32)<br>
self.label_tensor = self.label_tensor.flatten()<br>
# projected to 0, 18/1, 2/18, …, 17/18, 1<br>
<strong>self.label_tensor = self.label_tensor.div(18.0)</strong><br>
print(f"self.label_tensor.size() = {self.label_tensor.size()}")<br>
def <strong>len</strong>(self):<br>
‘Denotes the total number of samples’<br>
return len(self.data_tensor)<br>
def <strong>getitem</strong>(self, index):<br>
data, label = self.data_tensor[index], self.label_tensor[index]<br>
return data, label</h1>
<p>Then I reran the learning model, and found that the “nan” issue is gone, but I hit by another issue: after 100 epochs, all predicted values converged to the mean value, which is 0.5 and can be projected 11 in-advance-days. Please see the resultant output text below:<br>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
<strong>testing–&gt;pred = tensor([0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000])</strong><br>
testing–&gt;y = tensor([0.0556, 0.0556, 0.0556, 0.3889, 0.3889, 0.0556, 0.1111, 0.0556, 0.1111,<br>
0.2222])<br>
testing–&gt;loss = 0.13919752836227417<br>
testing–&gt;pred.argmax(0) = 0<br>
testing–&gt;(pred.argmax(0) == y) = tensor([False, False, False, False, False, False, False, False, False, False])<br>
testing–&gt;corr = 0.0<br>
<strong>testing–&gt;pred = tensor([0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000])</strong><br>
testing–&gt;y = tensor([0.1111, 0.2222, 0.3333, 0.0556, 0.1111, 0.4444, 0.0000, 0.0556, 0.1111,<br>
0.1111])<br>
testing–&gt;loss = 0.1358024775981903<br>
testing–&gt;pred.argmax(0) = 0<br>
testing–&gt;(pred.argmax(0) == y) = tensor([False, False, False, False, False, False,  True, False, False, False])<br>
testing–&gt;corr = 1.0<br>
<strong>testing–&gt;pred = tensor([0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000])</strong><br>
testing–&gt;y = tensor([0.1111, 0.1111, 0.0556, 0.1667, 0.1667, 0.1111, 0.0556, 0.2778, 0.0556,<br>
0.2778])<br>
testing–&gt;loss = 0.13672839105129242<br>
testing–&gt;pred.argmax(0) = 0<br>
testing–&gt;(pred.argmax(0) == y) = tensor([False, False, False, False, False, False, False, False, False, False])<br>
testing–&gt;corr = 0.0<br>
<strong>testing–&gt;pred = tensor([0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000])</strong><br>
testing–&gt;y = tensor([0.0000, 0.2222, 0.0000, 0.0556, 0.1111, 0.0556, 0.0556, 0.1111, 0.1111,<br>
0.0556])<br>
testing–&gt;loss = 0.18209877610206604<br>
testing–&gt;pred.argmax(0) = 0<br>
testing–&gt;(pred.argmax(0) == y) = tensor([ True, False,  True, False, False, False, False, False, False, False])<br>
testing–&gt;corr = 2.0<br>
<strong>testing–&gt;pred = tensor([0.5000, 0.5000, 0.5000, 0.5000])</strong><br>
testing–&gt;y = tensor([0.1667, 0.0556, 0.3333, 0.1667])<br>
testing–&gt;loss = 0.11188270896673203<br>
testing–&gt;pred.argmax(0) = 0<br>
testing–&gt;(pred.argmax(0) == y) = tensor([False, False, False, False])<br>
testing–&gt;corr = 0.0<br>
testing–&gt;test_loss = 1.3430555164813995, num_batches = 10<br>
testing–&gt;correct = 6.0, size = 94<br>
Test Error:<br>
Accuracy: 6.4%, Avg loss: 0.134306<br>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>Then the accuracy rate is only 6.4%.</p>
<p>Anybody can help me to diagnose this “Convergent to the same value” issue?</p>
<p>Thanks.</p>

Small_Red_69: <p>Right now I was running the learning model by 1000 epochs, but found that this time all predicted values are approaching to 0 (see the output text below).</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],<br>
pred = tensor([1.1067e-35, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([1.6154e-33, 1.6154e-33, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 7.7509e-16, 0.0000e+00,<br>
pred = tensor([0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0002,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0.0000e+00, 4.3497e-04, 1.0232e-17, 1.0232e-17, 1.0232e-17, 0.0000e+00,<br>
pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],<br>
pred = tensor([0.0000e+00, 4.8779e-15, 4.8779e-15, 4.8779e-15, 5.1095e-36, 0.0000e+00,<br>
pred = tensor([1.8333e-32, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 2.7634e-14,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 6.7904e-24, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 9.7125e-13, 1.5379e-35, 4.2071e-17,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 2.5491e-35, 0.0000e+00,<br>
pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],<br>
pred = tensor([0.0000e+00, 1.8329e-30, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0.0000e+00, 6.4101e-16, 0.0000e+00, 0.0000e+00, 1.2785e-36, 1.2785e-36,<br>
pred = tensor([1.1297e-18, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0.0000e+00, 5.8366e-35, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([1.1963e-12, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([7.0908e-31, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 9.5245e-36,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
pred = tensor([0., 0., 0., 0., 0., 0., 0.], grad_fn=)<br>
testing–&gt;pred = tensor([0.0000e+00, 1.2501e-15, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
testing–&gt;pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])<br>
testing–&gt;pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 1.8935e-14, 0.0000e+00,<br>
testing–&gt;pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])<br>
testing–&gt;pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])<br>
testing–&gt;pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])<br>
testing–&gt;pred = tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])<br>
testing–&gt;pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 1.2295e-15,<br>
testing–&gt;pred = tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 3.2930e-35,<br>
testing–&gt;pred = tensor([0., 0., 0., 0.])<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>So the trend is something like with the increase of learning epochs, the predicted values are all converged to 1, then keep on decreasing, and then after around 100 epochs, all of the predicted values are converged to 0.5, and finally all of them are converged and approached to 0.</p>
<p>But is this quite an abnormal learning results? I suspect that the functions employed by my NN are something wrong.</p>
<p>Can any experts help me to look into this “convergency” issue?</p>
<p>Thanks.</p>

ptrblck: <p><a class="mention" href="/u/kfrank">@KFrank</a> posted great tips in <a href="https://discuss.pytorch.org/t/why-does-my-deep-learning-model-get-only-30-cater-rate-even-after-1000-epochs/158194/6">this post</a> and advised against using a <code>Sigmoid</code> layer as it wouldn’t match your use case.<br>
Besides that he has also walked you through some steps what to check first and how to try to stabilize your training.</p>