Why does my deep-learning model (for classification problem) get all predicted values converge to a single label?

username: Small_Red_69
cooked: <p>Hi, friends,</p>
<p>Now I am exploring another PyTorch Deep-Learning model upon patients’ medical-appoint booking behaviours. This time it is to predict which weekday (from Monday to Friday) a patient will book a medical appointment. The range is 0 (for Monday) to 4 (for Friday).<br>
This is not a so complex user case. However, the accuracy rate of my model is still quite low: even after 1,000 epochs, the accuracy rate is only 25.3%, and the average loss is 1.608337.<br>
However, low accuracy rate is not the whole problem. To be worse, after debugging the deep-learning model, I found that somehow the output values of the model actually converges to 4 (Friday). It is understandable that the label 4 gets the highest percentage (is this due to a “relexable Friday” many people appreciate?) among the training data (26.2%) and the testing data (25.3%, that’s why the accuracy rate is 25.3%!), but all predicted results converging to a single label is still quite abnormal.<br>
The scenario of all-predicted-result-convert-to-a-single-label is shown in the output text below:<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
testing–&gt;pred = tensor([[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038]])<br>
testing–&gt;y = tensor([0, 2, 0, 2, 2, 2, 4, 1, 4, 0])<br>
testing–&gt;loss = 1.7076622247695923<br>
<strong>testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])</strong><br>
testing–&gt;(pred.argmax(1) == y) = tensor([False, False, False, False, False, False,  True, False,  True, False])<br>
testing–&gt;corr = 2.0<br>
testing–&gt;pred = tensor([[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038]])<br>
testing–&gt;y = tensor([0, 4, 4, 4, 1, 1, 4, 2, 1, 4])<br>
testing–&gt;loss = 1.4987714290618896<br>
<strong>testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])</strong><br>
testing–&gt;(pred.argmax(1) == y) = tensor([False,  True,  True,  True, False, False,  True, False, False,  True])<br>
testing–&gt;corr = 5.0<br>
testing–&gt;pred = tensor([[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038],<br>
[-0.4787,  0.1151,  0.0034,  0.0423,  0.3038]])<br>
testing–&gt;y = tensor([2, 1, 2, 4, 3, 4, 0, 4, 4])<br>
testing–&gt;loss = 1.5375866889953613<br>
<strong>testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])</strong><br>
testing–&gt;(pred.argmax(1) == y) = tensor([False, False, False,  True, False,  True, False,  True,  True])<br>
testing–&gt;corr = 4.0<br>
Test Error:<br>
Accuracy: 25.3%, Avg loss: 1.608337<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
In my model, the data tensor is a vector composed of the following 7 fields:</p>
<ol>
<li>Gender: 1 – male, 2 – female</li>
<li>Age</li>
<li>Area: I take the first 3 digits of a patient’s residential postal codes and map to an integer from 0 to 999.</li>
<li>Medical Examination: 1 – yes, 0 – no</li>
<li>Blood Test: 1 – yes, 0 – no</li>
<li>Urine Test: 1 – yes, 0 – no</li>
<li>Fasting: 1 – yes, 0 – no</li>
</ol>
<p>The labels of my model are from 0 (for Monday) to 4 (for Friday), totally 5 categories.<br>
Totally I prepare 397 training data and 99 testing data for the deep-learning model.</p>
<p>My deep-learning model has followed the Fashion-MNIST learning model from the tutorials in <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> official website (see the hyperlink <a href="https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optimizing Model Parameters — PyTorch Tutorials 1.12.0+cu102 documentation</a>). In the model, I also take the following parameters: the size of middle layer in the Neural Network is 512, the learning rate is 0.01 and the batch size is 10. (At the end of this post, I will attach the full set of my python source codes for your reference).</p>
<p>Can anybody help me to analyse and diagnose why all predicted values will converge to a single value? Is this due to the “overfitting” issue or the learning rate is still too high?</p>
<p>Thanks for the help. If you need more info to investigate the issue, please just let me know.</p>
<h1>
<a name="my-source-codes-of-the-pytorch-deep-learning-model-1" class="anchor" href="#my-source-codes-of-the-pytorch-deep-learning-model-1"></a>My source codes of the Pytorch deep-learning model:</h1>
<p>import torch<br>
from torch import nn<br>
from torch.utils.data import Dataset, DataLoader<br>
from torchvision import datasets<br>
from torchvision.transforms import ToTensor, Lambda<br>
import numpy as np<br>
import pandas as pd</p>
<p>class CustomDataset(Dataset):<br>
‘Characterizes a dataset for PyTorch’<br>
def <strong>init</strong>(self, csv_file):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file.<br>
“”"<br>
raw_data = pd.read_csv(csv_file)<br>
raw_data = torch.tensor(raw_data.to_numpy())<br>
x_size = list(raw_data.size())[1]<br>
self.data_tensor = raw_data[:,:x_size-1].clone()<br>
self.data_tensor = self.data_tensor.type(torch.float32)<br>
self.label_tensor = raw_data[:,x_size-1:].clone()<br>
self.label_tensor = self.label_tensor.flatten()<br>
print(f"self.label_tensor.size() = {self.label_tensor.size()}")<br>
def <strong>len</strong>(self):<br>
‘Denotes the total number of samples’<br>
return len(self.data_tensor)<br>
def <strong>getitem</strong>(self, index):<br>
data, label = self.data_tensor[index], self.label_tensor[index]<br>
return data, label</p>
<p>training_data_csv_file = “D:\Tools\PyTorch\Medex-Deep-Learning-Model\input_data\training_data.csv”<br>
training_data = CustomDataset(training_data_csv_file)<br>
test_data_csv_file = “D:\Tools\PyTorch\Medex-Deep-Learning-Model\input_data\testing_data.csv”<br>
test_data = CustomDataset(test_data_csv_file)</p>
<p>train_dataloader = DataLoader(training_data, batch_size=10)<br>
test_dataloader = DataLoader(test_data, batch_size=10)</p>
<p>class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 512),<br>
nn.ReLU(),<br>
nn.Linear(512, 5),<br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>model = NeuralNetwork()</p>
<p>def train_loop(dataloader, model, loss_fn, optimizer):<br>
size = len(dataloader.dataset)<br>
print(f’size = {size}‘);<br>
for batch, (X, y) in enumerate(dataloader):<br>
# Compute prediction and loss<br>
pred = model(X)<br>
print(f’pred = {pred}’);<br>
print(f’y = {y}‘);<br>
loss = loss_fn(pred, y)<br>
print(f’loss = {loss}’);<br>
# Backpropagation<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>    if batch % 100 == 0:
        loss, current = loss.item(), batch * len(X)
        print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>learning_rate = 0.01<br>
batch_size = 10</p>
<p>def test_loop(dataloader, model, loss_fn):<br>
size = len(dataloader.dataset)<br>
num_batches = len(dataloader)<br>
test_loss, correct = 0, 0</p>
<pre><code>with torch.no_grad():
    for X, y in dataloader:
        pred = model(X)
        print(f'testing--&gt;pred = {pred}');
        print(f'testing--&gt;y = {y}');
        loss = loss_fn(pred, y).item()
        print(f'testing--&gt;loss = {loss}');
        test_loss += loss
        print(f'testing--&gt;pred.argmax(1) = {pred.argmax(1)}');
        print(f'testing--&gt;(pred.argmax(1) == y) = {(pred.argmax(1) == y)}');
        corr = (pred.argmax(1) == y).type(torch.float).sum().item()
        print(f'testing--&gt;corr = {corr}');
        correct += corr
test_loss /= num_batches
correct /= size
print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")
</code></pre>
<p>loss_fn = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<h1>
<a name="epochs-1000-for-t-in-rangeepochs-printfepoch-t1n-train_looptrain_dataloader-model-loss_fn-optimizer-test_looptest_dataloader-model-loss_fn-printdone-2" class="anchor" href="#epochs-1000-for-t-in-rangeepochs-printfepoch-t1n-train_looptrain_dataloader-model-loss_fn-optimizer-test_looptest_dataloader-model-loss_fn-printdone-2"></a>epochs = 1000<br>
for t in range(epochs):<br>
print(f"Epoch {t+1}\n-------------------------------")<br>
train_loop(train_dataloader, model, loss_fn, optimizer)<br>
test_loop(test_dataloader, model, loss_fn)<br>
print(“Done!”)</h1>

username: Small_Red_69
cooked: <p>Suddenly I come to understand what the issue could be: this may caused by the argmax(1) function. This is because after 1000 epochs, each element of the predicted output tensor has converged to a constant. At this point, when I use the argmax(1) function, then the index of the largest value, which is 4, will always be selected at the moment, that’s why all the predicted values will be converged to the label 4.</p>
<p>However, instead of argmax(4), which function(s) can I use to cater to the labels from the testing data more accurately?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p><code>torch.argmax</code> will just return the index of the largest logit in your model’s output, i.e. the predicted class label in your case, and is not at fault.<br>
Your model seems to overfit to class4 in this run so you would need to check why that’s the case. E.g. maybe you are dealing with an imbalanced dataset where class4 is the majority class.</p>

username: Small_Red_69
cooked: <p>Hi, ptrblck,</p>
<p>you mentioned “maybe you are dealing with an imbalanced dataset where class4 is the majority class”.</p>
<p>But sorry, no, this is not the scenario. Actually my learning case is predict which weekday (from Monday to Friday) a patient is going to book an appointment, based on the gender, the age, the residential area, and the checking items (either a common medical checkup or a blood test or a urine test or an ECG test or multiple of them) of the patient.</p>
<p>Actually class4 is cast to Friday, from my training data, there are 126 out of 488 cases for a Friday appointment, occupying 25.8%. It is somewhat understandable that Friday cases are significantly more than other weekdays (maybe because many people may feel that Friday is quite relaxed?), but they don’t take the majority.</p>
<p>I can show you the distribution of all weekdays in the training data:<br>
Monday: 11.9%<br>
Tuesday: 21.7%<br>
Wednesday: 21.3%<br>
Thursday: 19.3%<br>
Friday: 25.8%</p>
<p>So you can see that the training dataset is not imbalanced.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You are right that the class distribution is not heavily imbalanced, but still <code>Friday</code> is the majority class (i.e. class with the largest number of samples) and your model seems to overfit to it.</p>

username: IneedMrmeeseeks
cooked: <p>Can you show us the training loss graph for each epoch?<br>
And I’d like to suggest that you need to do nomalization when you make CustomDataset class.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji only-emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Just now I got the training loss graph and want to show here:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54618ef3a25d62155e8dd750d8012c95005e354f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/54618ef3a25d62155e8dd750d8012c95005e354f" title="training_loss_graph"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54618ef3a25d62155e8dd750d8012c95005e354f_2_578x500.jpeg" alt="training_loss_graph" data-base62-sha1="c2t7Mb3b6VroeVXDZa2i7S3458X" width="578" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54618ef3a25d62155e8dd750d8012c95005e354f_2_578x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/4/54618ef3a25d62155e8dd750d8012c95005e354f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/4/54618ef3a25d62155e8dd750d8012c95005e354f.jpeg 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training_loss_graph</span><span class="informations">636×550 24.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please kindly review it.</p>
<p>Also can you please show me the normalization details when making CustomDataset class?</p>
<p>Thank you.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Right now I double-checked my training data, and found that the labels are already normalized to be within the range 0 to 4 (please see the blue rectangle in the screenshot below).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79512b6c751bbdfa5c34e5378f94bcdc0d06d4dd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/79512b6c751bbdfa5c34e5378f94bcdc0d06d4dd" title="training_data_label_already_normalized_within_range"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79512b6c751bbdfa5c34e5378f94bcdc0d06d4dd_2_690x374.jpeg" alt="training_data_label_already_normalized_within_range" data-base62-sha1="hjdG6eumTu6DuCaoaIBZ1pn6v25" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79512b6c751bbdfa5c34e5378f94bcdc0d06d4dd_2_690x374.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79512b6c751bbdfa5c34e5378f94bcdc0d06d4dd_2_1035x561.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/9/79512b6c751bbdfa5c34e5378f94bcdc0d06d4dd.jpeg 2x" data-dominant-color="F2F3F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training_data_label_already_normalized_within_range</span><span class="informations">1302×707 190 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can you please highlight what else I can do the normalize the CustomDataset class?</p>
<p>For your reference, I would like to show the source codes of my CustomDataset class below.</p>
<h1><a name="class-customdatasetdataset-characterizes-a-dataset-for-pytorch-def-__init__self-csv_file-args-csv_file-string-path-to-the-csv-file-raw_data-pdread_csvcsv_file-raw_data-torchtensorraw_datato_numpy-x_size-listraw_datasize1-selfdata_tensor-raw_datax_size-1clone-selfdata_tensor-selfdata_tensortypetorchfloat32-selflabel_tensor-raw_datax_size-1clone-selflabel_tensor-selflabel_tensorflatten-printfselflabel_tensorsize-selflabel_tensorsize-def-__len__self-denotes-the-total-number-of-samples-return-lenselfdata_tensor-def-__getitem__self-index-data-label-selfdata_tensorindex-selflabel_tensorindex-return-data-label-1" class="anchor" href="#class-customdatasetdataset-characterizes-a-dataset-for-pytorch-def-__init__self-csv_file-args-csv_file-string-path-to-the-csv-file-raw_data-pdread_csvcsv_file-raw_data-torchtensorraw_datato_numpy-x_size-listraw_datasize1-selfdata_tensor-raw_datax_size-1clone-selfdata_tensor-selfdata_tensortypetorchfloat32-selflabel_tensor-raw_datax_size-1clone-selflabel_tensor-selflabel_tensorflatten-printfselflabel_tensorsize-selflabel_tensorsize-def-__len__self-denotes-the-total-number-of-samples-return-lenselfdata_tensor-def-__getitem__self-index-data-label-selfdata_tensorindex-selflabel_tensorindex-return-data-label-1"></a>======================================================================<br>
class CustomDataset(Dataset):<br>
‘Characterizes a dataset for PyTorch’<br>
def <strong>init</strong>(self, csv_file):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file.<br>
“”"<br>
raw_data = pd.read_csv(csv_file)<br>
raw_data = torch.tensor(raw_data.to_numpy())<br>
x_size = list(raw_data.size())[1]<br>
self.data_tensor = raw_data[:,:x_size-1].clone()<br>
self.data_tensor = self.data_tensor.type(torch.float32)<br>
self.label_tensor = raw_data[:,x_size-1:].clone()<br>
self.label_tensor = self.label_tensor.flatten()<br>
print(f"self.label_tensor.size() = {self.label_tensor.size()}")<br>
def <strong>len</strong>(self):<br>
‘Denotes the total number of samples’<br>
return len(self.data_tensor)<br>
def <strong>getitem</strong>(self, index):<br>
data, label = self.data_tensor[index], self.label_tensor[index]<br>
return data, label</h1>
<p>Thanks.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Right now I tried to customize my learning model by decreasing the learning rate from 0.01 to 0.001. After running 1000 epochs, I can get a nearly-perfect training-loss decrement curve (as shown below):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08926d6263717f263d61afb9a3d9f0a4b6999af0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/08926d6263717f263d61afb9a3d9f0a4b6999af0" title="learning_rate_0.001_function_loss_curve"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08926d6263717f263d61afb9a3d9f0a4b6999af0_2_579x499.jpeg" alt="learning_rate_0.001_function_loss_curve" data-base62-sha1="1dPxp1SgqkEhuMo5CoefPKETou4" width="579" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08926d6263717f263d61afb9a3d9f0a4b6999af0_2_579x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/8/08926d6263717f263d61afb9a3d9f0a4b6999af0.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/8/08926d6263717f263d61afb9a3d9f0a4b6999af0.jpeg 2x" data-dominant-color="F6F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">learning_rate_0.001_function_loss_curve</span><span class="informations">642×554 43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Moreover, the cater rate has also increased from 30.9% to 41.4%. However, the predicted tensor values are still somehow converged to label 4 (see the output text below):</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
testing–&gt;corr = 5.0<br>
testing–&gt;pred = tensor([[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[ 1.1050, -1.1411,  0.8514,  0.0696,  0.6602]])<br>
testing–&gt;y = tensor([0, 0, 0, 0, 4, 0, 4, 4, 3, 2])<br>
testing–&gt;loss = 1.6169509887695312<br>
<strong>testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 0])</strong><br>
testing–&gt;(pred.argmax(1) == y) = tensor([False, False, False, False,  True, False,  True,  True, False, False])<br>
testing–&gt;corr = 3.0<br>
testing–&gt;pred = tensor([[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451],<br>
[-0.3121, -0.1847, -0.0434,  0.0043,  0.4451]])<br>
testing–&gt;y = tensor([1, 3, 4, 4, 4, 2, 2, 4, 0])<br>
testing–&gt;loss = 1.4929860830307007<br>
<strong>testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])</strong><br>
testing–&gt;(pred.argmax(1) == y) = tensor([False, False,  True,  True,  True, False, False,  True, False])<br>
testing–&gt;corr = 4.0<br>
Test Error:<br>
<strong>Accuracy: 41.4%</strong>, Avg loss: 1.523919</p>
<p>Done!<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>Do you have any good approaches to resolve the “convergency” issue?</p>
<p>Thanks.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed &amp; ptrblck,</p>
<p>Just now I reduced the #  of neurons in the middle-layer from 512 to 10 (see my NN class source codes below):</p>
<p>====================================================================<br>
class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
<strong>nn.Linear(7, 10),</strong><br>
nn.ReLU(),<br>
<strong>nn.Linear(10, 10),</strong><br>
nn.ReLU(),<br>
<strong>nn.Linear(10, 5),</strong><br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>====================================================================</p>
<p>I re-ran 1000 epochs, then got a super-perfect training-loss curve (see the diagram below)! Quite amazingly!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c7e008f1501d81387277c2e5ac3e4d2a030f26c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7c7e008f1501d81387277c2e5ac3e4d2a030f26c" title="training_loss_curve_with_10_neurons"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c7e008f1501d81387277c2e5ac3e4d2a030f26c_2_579x499.jpeg" alt="training_loss_curve_with_10_neurons" data-base62-sha1="hLjadmTsW5z2maVrZ3vEPbfENHK" width="579" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c7e008f1501d81387277c2e5ac3e4d2a030f26c_2_579x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c7e008f1501d81387277c2e5ac3e4d2a030f26c.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c7e008f1501d81387277c2e5ac3e4d2a030f26c.jpeg 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training_loss_curve_with_10_neurons</span><span class="informations">642×554 33.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, just the “convergency to label 4” issue is still pending. Please kindly help to give me some ideas on this issue.</p>
<p>Thanks!</p>

username: IneedMrmeeseeks
cooked: <p>Hi, I had national holiday yesterday.</p>
<p>You can customize your Dataset class. In getitem function, you can add some normalization lines.</p>
<p>Or you nomalize your dataframe first and then load the dataframe with this class. (you don’t need to normalize dependent variable.)<br>
It depends on your taste.<br>
Calculate mean and variance or Use scikit.preprocessing.</p>
<p>Also I recommend to use nn.softmax at the end of layers for interpretability.</p>
<p>Oops! I forgot to mention that you also need to preprocess the categorical variables.(binary 0, 1 will be fine. Gender should be changed to be 0, 1.)(Look up one-hot encoding or embedding)<br>
Normalization is for the continuous variables.</p>

username: IneedMrmeeseeks
cooked: <p>And I want to notify you that reducing learning rates is irrelevant with your issue.<br>
It may seems perfect loss curve, but It actually shows same loss value.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Thanks a lot for your feedback.<br>
May I keep on clarifying with you something?</p>
<p>“Also I recommend to use nn.softmax at the end of layers for interpretability.”</p>
<p>Do you mean that in the output layer, I will apply the nn.softmax function to normalize the predicted tensors? An example of a predicted tensor is shown below:</p>
<p>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
testing–&gt;pred = tensor([[-0.7003, -0.8617, -0.5095, -0.5607, -0.0856],<br>
[-1.8335, -1.4326, -1.3105, -1.2570, -0.6620],<br>
[-1.0715, -1.0413, -0.7767, -0.7803, -0.2806],<br>
[-0.5288, -0.8138, -0.3624, -0.4996,  0.0346],<br>
[-0.9010, -0.9953, -0.6293, -0.7216, -0.1595],<br>
[-1.1588, -1.1293, -0.8057, -0.8878, -0.2828],<br>
[-0.3873, -0.7340, -0.2692, -0.4018,  0.0979],<br>
[-0.6543, -0.8487, -0.4722, -0.5421, -0.0564],<br>
[-0.8280, -0.9215, -0.6081, -0.6287, -0.1617]])<br>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>so do you mean that I need to perform the nn.Softmax(dim=1) operation upon the above-mentioned output tensors to normalize the final output tensors?</p>
<p>“I forgot to mention that you also need to preprocess the categorical variables.(binary 0, 1 will be fine. Gender should be changed to be 0, 1.)”</p>
<p>Yes, I will normalize the gender values as 0 and 1. However, I also define the other categorical variables (Age and Area) as discrete variables:</p>
<p>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
<strong>Age</strong>: from 18 to 63, totally 46 possible values<br>
<strong>Area</strong>: from 000 to 999, totally 1000 possible values<br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>So please show me whether I also need to normalize the Age and Area variables to be within the range [0, 1], for example, regarding to an age values, I can cater 18 to value 0, cater 19 to value 0+1/45=0.022, so on and so forth, finally cater 63 to value 1. For Area values, I will cater 000 to 0.0, 001 to 0.001, so on and so forth, then cater 999 to 0.999.</p>
<p>Can I do things by this way?</p>
<p>Thanks.</p>

username: IneedMrmeeseeks
cooked: <p>Actually nn.CrossEntropyLoss does the logsoftmax for you. Here is the <a href="https://discuss.pytorch.org/t/softmax-cross-entropy-loss/125383/2">link</a><br>
So, you don’t need to add a softmax layer. sry for the confusion.</p>
<p>Age, Area should be considered as continuous variables. That’s why I mentioned about normalization.<br>
Your last paragraph explanation is so called “Min-Max normalization”.<br>
I talked about more like standardization which use mean and variance(mean=0 and std=1).<br>
Both ways would be fine.</p>
<p>edit) Sry again. Area is the post code!! Ok, then you may want to use <code>embedding</code> with fixed length.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Sorry what do you mean " you may want to use <code>embedding</code> with fixed length"? What is “embedding”?</p>
<p>Thanks.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>I have almost tried all your proposed methods to address the “convergency to same label value” issue.</p>
<ol>
<li>“So, you don’t need to add a softmax layer”</li>
</ol>
<p>I already removed the software function from the NN structure. Please see the source codes of my NN class below:</p>
<p>=====================================================================<br>
class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(7, 10),<br>
nn.ReLU(),<br>
nn.Linear(10, 10),<br>
nn.ReLU(),<br>
nn.Linear(10, 5),<br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>=====================================================================</p>
<ol start="2">
<li>“Min-Max normalization”<br>
I have implemented the “Min-Max normalization” upon the Gender, Age and Area fields, making all of their values within the range of [0, 1]: In the Gender field, I subtract all values by 1, making a male patient to get the value 1, and a female patient to get the value 0; In the Age field, I divided all values by 100, making the age values between 0.18 and 0.63; In the Area field, since there are 6-digit postal codes, at first I take the first 3 digits, to make the range between 0 and 999, and then divided them by 1000, making the area values between 0 and 1.<br>
Please refer to the screenshot below to see my normalization result of the input data.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2ecd7d8ff501631a0276aeb31b331d148732538.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d2ecd7d8ff501631a0276aeb31b331d148732538" title="Already_Performed_Min_Max_Normalization_Upon_Gender_Age_Area_fields"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2ecd7d8ff501631a0276aeb31b331d148732538_2_690x387.jpeg" alt="Already_Performed_Min_Max_Normalization_Upon_Gender_Age_Area_fields" data-base62-sha1="u5VHzZVHM9SKAkodpHyCFjgSVcY" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2ecd7d8ff501631a0276aeb31b331d148732538_2_690x387.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2ecd7d8ff501631a0276aeb31b331d148732538_2_1035x580.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2ecd7d8ff501631a0276aeb31b331d148732538.jpeg 2x" data-dominant-color="E4E5E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Already_Performed_Min_Max_Normalization_Upon_Gender_Age_Area_fields</span><span class="informations">1366×768 227 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ol>
<p>Thereafter I run 1000 epochs again. However, to be frustrated, all the predicted values are still converged to label 4. (see the output text below)</p>
<p>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
testing–&gt;pred.argmax(1) = tensor([4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>Do you have any more approaches to resolve the “convergency” issue?</p>
<p>Thanks.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Actually I want to share with you the full set of my training data &amp; testing data, but don’t know how to upload csv files in this forum.</p>
<p>So far the size of the training data is only 118, and the testing data is only 29. Can the pending “convergency” issue be caused by these too small size training &amp; testing dataset?</p>
<p>Please kindly help me to diagnose the issue.</p>
<p>Later on I will try to find a way to share my training/testing dataset with you.</p>

username: IneedMrmeeseeks
cooked: <p>Hi, there.<br>
I have my own work to do. It’d be better if you look up the way to solve your problem.<br>
I want you to know the way to solve, not just the answer.</p>
<p>Here is the <a href="https://towardsdatascience.com/deep-learning-for-tabular-data-using-pytorch-1807f2858320" rel="noopener nofollow ugc">article</a> which is very similar with your  problem. Take a look and I hope this article helps you.</p>
<p>Best regard.</p>

username: Small_Red_69
cooked: <p>Hi, Ineed,</p>
<p>Just now I uploaded my DataSet csv files into Microsoft One-Drive for your reference.</p>
<p><strong>Training Date</strong></p>
<p><a href="https://onedrive.live.com/Edit.aspx?resid=8FFA4A1C66CFCED9!1161&amp;wd=cpe" rel="noopener nofollow ugc">training_data.xlsx - Microsoft Excel Online (live.com)</a></p>
<p><strong>Testing Data</strong></p>
<p><a href="https://onedrive.live.com/Edit.aspx?resid=8FFA4A1C66CFCED9!1160&amp;wd=cpe" rel="noopener nofollow ugc">testing_data.xlsx - Microsoft Excel Online (live.com)</a></p>
<p>It is ok that you are concentrated on your work at 1st. If only u get time, then you can help to  view my case. No hurry.</p>
<p>Thank you for the help and sorry for any inconvenience caused.</p>