My loss gets to 0 although the expected and predicted values doesnt match, does anyone know why?

hananelroe: <p>here is my code, I’m just trying to diffrentiate between two .wav files, for exercise. the final target for the project is a hotword detector:</p>
<pre><code class="lang-auto">import torch
import torchaudio
import matplotlib.pyplot as plt
import os
import torch.nn as nn
import soundfile as sf
from pydub import AudioSegment

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")

AUDIO_DIR = './data/audio/'
IDX = 1
REUQESTED_FILE = str(IDX) + '.wav'
FilePath = os.path.join(str(AUDIO_DIR), str(REUQESTED_FILE))


def waveformToMelSpecrogram(wavFilePath):
    waveform, sample_rate = torchaudio.load(wavFilePath)
    waveform, sample_rate = sf.read(wavFilePath)
    waveform = AudioSegment.from_mp3(wavFilePath)
    waveform = waveform.set_channels(1)
    waveform = waveform.get_array_of_samples()
    waveform = torch.tensor(waveform, dtype=torch.float)
    waveform = torch.reshape(waveform, (1, waveform.shape[0]))
    mel_spectrogram = torchaudio.transforms.MelSpectrogram()(waveform)
    return mel_spectrogram


class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(128 * 1756, 1500),
            nn.ReLU(),
            nn.Linear(1500, 1000),
            nn.ReLU(),
            nn.Linear(1000, 500),
            nn.ReLU(),
            nn.Linear(500, 2)
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits


specgram = waveformToMelSpecrogram(FilePath)

print(f"Shape of spectrogram: {specgram.size()}")

'''plt.figure()
plt.imshow(specgram.log2()[0,:,:].numpy())
plt.show()'''

net = NN().to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)


expected = torch.tensor([1])
print(f"expected: {expected}")

loss = 9999999999

while loss != 0:
    prediction = net(specgram)
    print(f"prediction: {prediction}")
    # Compute prediction error
    loss = loss_fn(prediction, expected)
    print(f"loss: {loss}")

    # Backpropagation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>the output:</p>
<pre><code class="lang-auto">C:\Users\hagit\AppData\Local\Microsoft\WindowsApps\python3.9.exe "C:/Users/hagit/OneDrive/שולחן העבודה/חננאל/neural networks/pytorch tutorials/speech recognition.py"
C:\Users\hagit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pydub\utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
Using cpu device
C:\Users\hagit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\torchaudio\functional\functional.py:432: UserWarning: At least one mel filterbank has all zero values. The value for `n_mels` (128) may be set too high. Or, the value for `n_freqs` (201) may be set too low.
  warnings.warn(
Shape of spectrogram: torch.Size([1, 128, 1756])
expected: tensor([1])
prediction: tensor([[ 1.1202e+09, -2.1888e+09]], grad_fn=&lt;AddmmBackward&gt;)
loss: 3309037824.0
prediction: tensor([[-1.0482e+12,  1.1267e+12]], grad_fn=&lt;AddmmBackward&gt;)
loss: 0.0

Process finished with exit code 0

</code></pre>

KFrank: <p>Hi hananelroe!</p>
<aside class="quote no-group quote-modified" data-username="hananelroe" data-post="1" data-topic="133845" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hananelroe/48/41298_2.png" class="avatar"> hananelroe:</div>
<blockquote>
<pre><code class="lang-auto">loss_fn = nn.CrossEntropyLoss()
...
    loss = loss_fn(prediction, expected)
    print(f"loss: {loss}")
</code></pre>
<pre><code class="lang-auto">expected: tensor([1])
prediction: tensor([[ 1.1202e+09, -2.1888e+09]], grad_fn=&lt;AddmmBackward&gt;)
loss: 3309037824.0
prediction: tensor([[-1.0482e+12,  1.1267e+12]], grad_fn=&lt;AddmmBackward&gt;)
loss: 0.0
</code></pre>
</blockquote>
</aside>
<p>First, with <code>CrossEntropyLoss</code>, you don’t expect your predicted<br>
values (the <code>input</code> to <code>CrossEnropyLoss</code>) and the expected values<br>
(the <code>target</code> for <code>CrossEntropyLoss</code> to be the same.  They are,<br>
roughly speaking, encoded differently.</p>
<p>The <code>input</code> is a set of <code>nClass</code> <code>logits</code>, while the <code>target</code> is a single<br>
integer class label (that runs from <code>0</code> to <code>nClass - 1</code>).</p>
<p>Now, to your specific case:</p>
<p>When you say <code>expected</code> is <code>tensor ([1])</code>, you are saying that you<br>
have a batch of batch size 1 and that that single sample is in class<br>
“1”.</p>
<p>When you say that your first value for <code>prediction</code> is<br>
<code>tensor ([[ 1.1202e+09, -2.1888e+09]])</code>, you are saying that<br>
you have a (very) large positive logit for the sample being in class<br>
“0” and a (very) large negative logit for the sample being in class “1”.<br>
When you (conceptually) convert the logits to probabilities, you are<br>
predicting that the sample is in class “0” with a probability very<br>
close to <code>1.0</code>.  That is, you are making a highly certain <em>incorrect</em><br>
prediction and your loss is large.</p>
<p>Conversely, your second value for <code>prediction</code>,<br>
<code>tensor ([[-1.0482e+12,  1.1267e+12]])</code>, is the other way around,<br>
and corresponds to a prediction probability of very nearly <code>1.0</code> that<br>
your sample is in class “1”.  You are making a highly certain <em>correct</em><br>
prediction so your loss is very close to <code>0.0</code>.</p>
<p>Internal to <code>CrossEntropyLoss</code> is a <code>softmax()</code> function (actually<br>
<code>log_softmax()</code>) that converts the <code>input</code> logits into probabilities.</p>
<p>Let’s see what we get when we apply <code>softmax()</code> to your values<br>
for <code>prediction</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; prediction_1 = torch.tensor ([[ 1.1202e+09, -2.1888e+09]])
&gt;&gt;&gt; prediction_2 = torch.tensor ([[-1.0482e+12,  1.1267e+12]])
&gt;&gt;&gt; torch.nn.functional.softmax (prediction_1, dim = 1)
tensor([[1., 0.]])
&gt;&gt;&gt; torch.nn.functional.softmax (prediction_2, dim = 1)
tensor([[0., 1.]])
</code></pre>
<p>In fact, your logits are so large that within the numerical range<br>
of floating-point numbers, you get exactly <code>1.0</code> for the predicted<br>
probability of being in class “0” and “1”, respectively, hence the<br>
loss of exactly <code>0.0</code> for your second prediction.</p>
<p>Best.</p>
<p>K. Frank</p>

hananelroe: <p>then I sould apply softmax to the net output? the output it gives me is <code>tensor([[0., 1.]]</code>. what did i do wrong?<br>
sorry if this is a stupid question, this is my first ML project</p>