I'm getting 0% accuracy on every epoch while loss decreases

elemeo: <p>Hello everyone and thanks in advance for taking the time to read through my issue. I’m currently trying to tackle regression on a Time Series (Bitcoin prices in the last 5 years). To spare you all the boring details my dataset is as follows after preprocessing:</p>
<pre><code class="lang-auto">0       0.000846
1       0.001299
2       0.002159
3       0.001813
4       0.001773
          ...   
1822    0.680117
1823    0.689799
1824    0.687676
1825    0.690270
1826    0.715508
Name: Close, Length: 1827, dtype: float64
</code></pre>
<p>This is basically the Closing price for every day in the past 5 years (its normalized based on the entire dataset then extracted). From this, I want to create a “sliding window” dataset so I can use the previous 50 days to predict the price of the next day. My custom dataset is as follows:</p>
<pre><code class="lang-auto">class TimeSeriesDataset(Data.Dataset):   
    def __init__(self, data, window):
        self.data = torch.Tensor(data.values)
        self.window = window
    
    def __getitem__(self, index):
        return (self.data[index:index+self.window], self.data[index+self.window])
    
    def __len__(self):
        return self.data.__len__() - (self.window)
</code></pre>
<p>I’ve manually split the data into training and testing (80% to training) and  my network architecture is this:</p>
<pre><code class="lang-auto">self.hidden1 = torch.nn.Linear(50, 25) # hidden layer
self.hidden2 = torch.nn.Linear(25, 25) # hidden layer
self.out = torch.nn.Linear(25, 1)     # output layer
</code></pre>
<p>and these are my training and testing methods:</p>
<pre><code class="lang-auto">def train(dataset, model, loss_fn, optimizer):
    size = len(dataset)
    model.train()
    for batch, (X, y) in enumerate(dataset):
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X).squeeze(-1)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch
            print(f"loss: {loss:&gt;7f}  Batch: [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<pre><code class="lang-auto">def test(dataset, model, loss_fn):
    size = len(dataset)
    num_batches = len(dataset) - 50
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataset:
            X, y = X.to(device), y.to(device)
            pred = model(X).squeeze(-1)
            test_loss += loss_fn(pred, y).item()
            testing_acc = torch.sum(pred == y)
    test_loss /= num_batches
    print(f"Test Error: \n Accuracy: {testing_acc}%, Avg loss: {test_loss:&gt;8f} \n")
</code></pre>
<p>If I now run this for 20 epochs I notice that on every epoch the loss drops really low, however my accuracy is always 0% because none of my predicted values match the labels.</p>
<p>I figured that I wouldnt be able to predict the price with good accuracy (because if I could then I could beat the market I guess ?) but 0% seems way too low (Or that something’s wrong) especially when loss is dropping. Am I calculating this the wrong way ? What can I try ?<br>
Thanks again and sorry for the long post.</p>

KFrank: <p>Hi Elemeo!</p>
<aside class="quote no-group quote-modified" data-username="elemeo" data-post="1" data-topic="140641" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elemeo/48/33155_2.png" class="avatar"> elemeo:</div>
<blockquote>
<pre><code class="lang-auto">self.out = torch.nn.Linear(25, 1)     # output layer
</code></pre>
<pre><code class="lang-auto">            testing_acc = torch.sum(pred == y)
</code></pre>
<p>my accuracy is always 0% because none of my predicted values match the labels.</p>
</blockquote>
</aside>
<p>You are testing for the <em>exact</em> equality of floating-point numbers.  In<br>
general (except in cases of “special” values like 0.0) two floating-point<br>
numbers, even if very nearly equal, are <em>extremely</em> unlikely to be exactly<br>
equal.  Therefore it is expected that you never have a “correct” prediction<br>
and that your computed accuracy is 0% (even if your predictions are<br>
actually very good).</p>
<aside class="quote no-group">
<blockquote>
<p>I figured that I wouldnt be able to predict the price with good accuracy (because if I could then I could beat the market I guess ?)</p>
</blockquote>
</aside>
<p>As an aside, it is true that predicting tradable market prices is a difficult<br>
problem.  (To first approximation, the best prediction for today’s closing<br>
price will be yesterday’s closing price.)  If you think about it, many<br>
challenging neural-network problems – say, distinguishing cats from dogs<br>
or localizing all of the instances of people in an image – are altogether<br>
quite easy for people to do.  But ask yourself how easy it is for you to<br>
predict today’s Bitcoin closing price (more accurately than just using<br>
yesterday’s closing price) by looking at the time series up through<br>
yesterday.  That should suggest that this could be quite hard for a neural<br>
network, as well.</p>
<p>Best.</p>
<p>K. Frank</p>

elemeo: <p>I understand that absolutely and I don’t really expect anything amazing to come out of it, it’s more like an exercise and a way to get familiar with PyTorch.</p>
<p>Regarding floating points I figured that it would be almost impossible to get exact numbers with 2 decimals so accuracy will always be 0%.</p>
<p>Do you have any suggestions as to what I would use to measure accuracy ?</p>
<p>Thank you.</p>

KFrank: <p>Hi Elemeo!</p>
<aside class="quote no-group" data-username="elemeo" data-post="3" data-topic="140641" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elemeo/48/33155_2.png" class="avatar"> elemeo:</div>
<blockquote>
<p>I figured that it would be almost impossible to get exact numbers with 2 decimals so accuracy will always be 0%.</p>
</blockquote>
</aside>
<p>Just to be clear, in your above code you’re not checking whether the<br>
floating-point numbers agree to 2 decimal digits of accuracy, but rather<br>
whether they agree exactly, which is to say to about 7 decimal digits of<br>
accuracy.</p>
<aside class="quote no-group">
<blockquote>
<p>Do you have any suggestions as to what I would use to measure accuracy ?</p>
</blockquote>
</aside>
<p>A very reasonable measure of accuracy could be the mean-squared-error<br>
between your predictions and your targets.  (This could also be a good<br>
choice for the loss function you use for training.)</p>
<p>If your data are market prices given in, say, dollars, the units of the<br>
mean-squared-error will be dollars-squared.  So it might be more intuitive<br>
to repackage the same information as the root-mean-squared-error.<br>
Thus, using the variables from the code you posted:</p>
<pre data-code-wrap="python"><code class="lang-python">(pred - y).mean().sqrt()
</code></pre>
<p>This will have units of dollars, which might be a little easier to think about.</p>
<p>Note, however, that your time series starts of very small and grows over<br>
time.  This means that errors in your predictions will likely be much smaller<br>
and count much less early on rather than later.  For this reason you may<br>
prefer to use fractional errors, e.g., something like <code>(pred - y) / y</code>.</p>
<p>If it’s important that your use case involve predicting tradable market<br>
prices, then bear in mind that is it is relatively easy, but uninteresting,<br>
to predict the price itself – just use yesterday’s price as your prediction.<br>
Much more interesting – and also much harder – is to predict the<br>
change in price.</p>
<p>I would suggest that you try to predict fractional price changes (“returns”),<br>
or, perhaps more elegantly, <em>logarithmic</em> returns:</p>
<pre data-code-wrap="python"><code class="lang-python">r_t  =  log (p_t / p_(t - 1))
</code></pre>
<p>where <code>p_t</code> is the price at time <code>t</code>, and so on.</p>
<p>The mean-squared-error, <code>((r_pred - r_targ)**2).mean()</code> would be<br>
both a sensible loss function for training and accuracy metric for<br>
evaluation.</p>
<p>If you want to get fancy, you could even use the results of a (differentiable)<br>
profit-and-loss trading “simulation” as your training loss and/or evaluation<br>
metric.  That is, you take a “position” based on how large your predicted<br>
return is.  If your prediction is in the right direction, you make a “profit”<br>
on that position, and if it’s wrong, you take a “loss.”</p>
<p>Best.</p>
<p>K. Frank</p>

elemeo: <p>Will try, thank you for the suggestions.</p>