Accuracy doesn't change

NiLSPACE: <p>Hi all,</p>
<p>The model that I’m trying to train doesn’t seem to increase it’s accuracy. I’m used to the MLPClassifier from scikit-learn because it’s what we used in our datascience classes, but I was hoping to use PyTorch to create more advanced models. I’m using the Skorch library wrapper to remove allot of boiler plate code.</p>
<h2><a name="purpose-of-the-model-1" class="anchor" href="#purpose-of-the-model-1"></a>Purpose of the model</h2>
<p>I’m working with radar data of road surfaces. In the data there are lots of echo’s of the road surface. The purpose of the model is to detect the actual road. When converting a slice of the data to an image it looks like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fd1e9fe8007d36fc6fce66bad8007eac4b19177.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fd1e9fe8007d36fc6fce66bad8007eac4b19177" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd1e9fe8007d36fc6fce66bad8007eac4b19177_2_51x499.png" alt="image" data-base62-sha1="kwi13sHJfmVMmwLIkKjirFkReRN" width="51" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd1e9fe8007d36fc6fce66bad8007eac4b19177_2_51x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fd1e9fe8007d36fc6fce66bad8007eac4b19177_2_76x748.png 1.5x" data-dominant-color="787878"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">76×734 12.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The red line is what Scikit’s MLPClassifier predicts and the green dots are the actual road surfaces which were manually input to train the dataset. With the MLPClassifier I was able to use the position as the target label. The MLPClassifier worked on a per-project basis, but got too slow when trying to combine all the projects together. I was hoping to use PyTorch which can use my GPU to create a more advanced model.</p>
<h2><a name="data-2" class="anchor" href="#data-2"></a>Data</h2>
<p>The data consists of two files. The input file and the prediction/label file. I’ve already processed them so each row correlate to each other.</p>
<h3><a name="input-file-3" class="anchor" href="#input-file-3"></a>Input file</h3>
<p>The input data is a CSV file where every row consists of 512 floating point numbers. The numbers range from roughly -5000 to 5000.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>0</th>
<th>1</th>
<th>…</th>
<th>510</th>
<th>511</th>
</tr>
</thead>
<tbody>
<tr>
<td>512.0</td>
<td>230.0</td>
<td></td>
<td>-1836.0</td>
<td>-1836.0</td>
</tr>
<tr>
<td>512.0</td>
<td>225.0</td>
<td></td>
<td>-1802.0</td>
<td>-1802.0</td>
</tr>
</tbody>
</table>
</div><h3><a name="label-file-4" class="anchor" href="#label-file-4"></a>Label file</h3>
<p>This file consists of two numbers, but only the Y-column is used. The values in this column are used as the target label.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>X</th>
<th>Y</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>211</td>
</tr>
<tr>
<td>3</td>
<td>212</td>
</tr>
</tbody>
</table>
</div><h2><a name="code-5" class="anchor" href="#code-5"></a>Code</h2>
<p>As I said earlier I’m using the Skorch library to remove allot of the boiler plate code. I have tried using PyTorch directly, but I had similar issues where each epoch didn’t change the accuracy. This makes me think either my model/layers are wrong or I’m making a mistake while providing the data to the model. The code I’m using looks like the following:</p>
<pre data-code-wrap="py"><code class="lang-py">import torch
import torch.nn as nn
import pandas as pd
import skorch
import torch.nn.functional as F

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Read input data
data = pd.read_csv("project-input.csv", index_col=None)
X = data

# Read prediction data
predictions = pd.read_csv("project-predictions.csv", index_col=None)
y = predictions['Y']

# The model
class Multiclass(nn.Module):
    def __init__(self, num_outputs=512):
        super(Multiclass, self).__init__()
        self.dropout = nn.Dropout(0.5)
        self.hidden=nn.Linear(512, 2048)
        self.output = nn.Linear(2048, 512)
        
    def forward(self, x):
        x = x.reshape(-1, self.hidden.in_features)
        x = F.relu(self.hidden(x))
        x = self.dropout(x)
        x = F.softmax(self.output(x), dim=-1)
        return x

net = skorch.classifier.NeuralNetClassifier(
    module=Multiclass,
    device=device
)


X = torch.tensor(X.values, dtype=torch.float32)
y = torch.tensor(y.values, dtype=torch.int64)

X = X.to(device)
y = y.to(device)

net.fit(X, y)
</code></pre>
<p>This runs, but the accuracy does not improve.</p>
<pre><code class="lang-auto">  epoch    train_loss    valid_acc    valid_loss      dur
-------  ------------  -----------  ------------  -------
      1        6.2400       0.0008        6.2409  20.5180
      2        6.2403       0.0008        6.2409  17.4280
      3        6.2403       0.0008        6.2409  18.2130
      4        6.2400       0.0008        6.2409  19.0680
      5        6.2402       0.0008        6.2409  20.5232
</code></pre>
<p>I was hoping there is an obvious mistake that people see. As I said earlier I’m mostly familiar with the MLPClassifier, so the most I’ve previously messed around with was the hidden layers.</p>

ptrblck: <p>The <code>NeuralNetClassifier</code> seems to use <code>nn.NLLLoss</code> as its default criterion so you would need to pass log probabilities to it via <code>F.log_softmax</code> instead of <code>F.softmax</code>. Change it and see if this would improve the PyTorch model training.</p>

NiLSPACE: <p>Thanks for the fast response!</p>
<p>I’ve tried replacing <code>softmax(self.output(x), dim=-1)</code> with <code>log_softmax(self.output(x), dim=-1)</code> and now it outputs</p>
<pre><code class="lang-auto">  epoch    train_loss    valid_acc    valid_loss      dur
-------  ------------  -----------  ------------  -------
      1           nan       0.0000           nan  10.8670
      2           nan       0.0000           nan  6.3860
      3           nan       0.0000           nan  5.3280
      4           nan       0.0000           nan  5.2080
      5           nan       0.0000           nan  5.6330
      6           nan       0.0000           nan  5.8010
</code></pre>

ptrblck: <p>Could you print the outputs before returning them from the <code>forward</code> to check if these are overflowing?</p>

NiLSPACE: <p>Adding a <code>print(x)</code> just before returning from <code>forward</code> results in the following:</p>
<pre><code class="lang-auto">tensor([[-1561.2009, -2532.7739,  -892.7590,  ..., -2790.2212, -2430.4778,
         -3008.7415],
        [-3527.6836, -3519.8684,  -970.9326,  ..., -2911.9658, -2074.8315,
         -2793.2590],
        [-2680.8403, -2953.5088, -2104.6016,  ..., -2319.5645, -1220.5420,
         -4527.9507],
        ...,
        [-1753.6694, -1107.4957, -2239.1433,  ..., -2213.6201, -2000.9958,
         -1967.8594],
        [-2916.8323, -3349.4568, -1327.4064,  ..., -2687.6819, -1950.6512,
         -3970.4326],
        [-1833.4132, -2364.8330, -1087.6481,  ..., -2808.8252,  -785.6394,
         -2791.5884]], device='cuda:0', grad_fn=&lt;LogSoftmaxBackward0&gt;)
</code></pre>
<p>I’m not familiar enough with PyTorch (or machine learning in general) to know if it’s overflowing. How can I recognize that?</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="NiLSPACE" data-post="1" data-topic="201923">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nilspace/48/69688_2.png" class="avatar"> NiLSPACE:</div>
<blockquote>
<pre><code class="lang-auto">y = torch.tensor(y.values, dtype=torch.int64)
</code></pre>
</blockquote>
</aside>
<p>Can you backprop if your output is an integer?</p>

ptrblck: <p>Are all values valid in this output and is <code>torch.isfinite(output).all()</code> returning <code>True</code>?</p>