Neural Network only gives outputs of 0

ApeelingPotato: <p>Hi there!</p>
<p>I am trying to train a neural network with a tensor of 1040 float inputs and have a singular float output. No matter what I do the predicted value comes out as 0<br>
I think the problem I am having is with teaching the network. My Net class is as below. The interneuron_count is 64.</p>
<pre><code class="lang-auto"> class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.fc1 = nn.Linear(1040, interneuron_count)  # Fully connected layer1 with 1040 inputs
            self.fc2 = nn.Linear(interneuron_count, interneuron_count)
            self.fc3 = nn.Linear(interneuron_count, 1)  # final fully connected layer with one output

        def forward(self, x):
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return F.log_softmax(x, dim=0)
</code></pre>
<p>This is the code segment on how I train my neural network:</p>
<pre><code class="lang-auto">    for epoch in range(EPOCHS):
        for i in tqdm(range(0, (len(train_inputs)), BATCH_SIZE)):
            batch_X = train_inputs[i:i+BATCH_SIZE]
            batch_Y = train_outputs[i:i+BATCH_SIZE]

            net.zero_grad()
            outputs = net(batch_X)
            loss = loss_function(outputs, batch_Y)
            loss.backward()

            if epoch &gt; learning_rate_cutoff:  # this acts as a decaying learning rate for our system
                learning_rate = 0.0001
            optimizer.step()
</code></pre>
<p>When I test the NN using     print(net(test_inputs[0]))  I get<br>
<code>tensor([0.], grad_fn=&lt;LogSoftmaxBackward&gt;)</code></p>
<p>instead of a value of around 1.25901</p>
<p>Any help with this would be greatly appreciated! Thank you</p>

KFrank: <p>Hi Brian!</p>
<aside class="quote no-group quote-modified" data-username="ApeelingPotato" data-post="1" data-topic="77250" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f07891/48.png" class="avatar"> ApeelingPotato:</div>
<blockquote>
<p>No matter what I do the predicted value comes out as 0<br>
I think the problem I am having is with teaching the network.</p>
<pre><code class="lang-auto">            self.fc3 = nn.Linear(interneuron_count, 1)  # final fully connected layer with one output
...
            x = self.fc3(x)
            return F.log_softmax(x, dim=0)
</code></pre>
<p>When I test the NN using     print(net(test_inputs[0]))  I get<br>
<code>tensor([0.], grad_fn=&lt;LogSoftmaxBackward&gt;)</code></p>
</blockquote>
</aside>
<p>Your problem is that you are passing a length-one vector to<br>
<code>softmax()</code>.  <code>softmax()</code> returns a vector of probabilities that<br>
sum to one.  Since there is only one probability in the sum, it<br>
will always be <code>1.0</code>.  <code>log (1.0) = 0.0</code>, so, analogously,<br>
<code>log_softmax()</code> will always return <code>0.0</code>.</p>
<p>If this network is for a binary classification problem, and your<br>
single output is supposed to indicate whether your input is in<br>
class-“0” or c;ass-“1”, then you should have</p>
<pre data-code-wrap="python"><code class="lang-python">    return F.sigmoid (x)
</code></pre>
<p>and use <code>BCELoss</code> for your loss function (or just <code>return x</code> without<br>
the <code>sigmoid()</code>, and use <code>BCEWithLogitsLoss</code>).</p>
<p>As an aside, in <code>return F.log_softmax(x, dim=0)</code>, <code>dim = 0</code> is<br>
the batch dimension.  I’m guessing in the example you gave that<br>
your batch size in 1.  If it did make sense to use <code>log_softmax()</code><br>
(which it doesn’t, but would, for example, if your last layer were<br>
<code>Linear(interneuron_count, 10)</code>) you would want something like<br>
<code>log_softmax(x, dim = 1)</code> or <code>log_softmax(x, dim = -1)</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

actuallyaswin: <p>To add to KFrank’s post, this deals with the fundamental difference between sigmoid and softmax. A more detailed comparison can be found <a href="https://stats.stackexchange.com/a/254071" rel="nofollow noopener">here on StackExchange</a>; specifically with regards to neural networks, <a href="https://stats.stackexchange.com/a/410112" rel="nofollow noopener">this post on StackExchange</a> is also helpful.</p>

ApeelingPotato: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="77250">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>return F.sigmoid (x)</p>
</blockquote>
</aside>
<p>Hi KFrank! Thanks for getting back to me</p>
<p>Will this return a float out of the neural network?</p>
<blockquote>
<p>return F.sigmoid (x)</p>
</blockquote>
<p>Im only getting 1s out of the NN.</p>
<p>Im using batch sizes of ten into the network. I also started with using the optim.Adam and the MSELoss function</p>

KFrank: <p>Hi Brian!</p>
<aside class="quote no-group quote-modified" data-username="ApeelingPotato" data-post="4" data-topic="77250" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f07891/48.png" class="avatar"> ApeelingPotato:</div>
<blockquote>
<p>return F.sigmoid (x)</p>
<p>Will this return a float out of the neural network?</p>
</blockquote>
</aside>
<p>If <code>x</code> is a <code>FloatTensor</code>, it will return a <code>FloatTensor</code>.  In general, it<br>
will return a tensor of the type passed in (if the type is supported).</p>
<aside class="quote no-group">
<blockquote>
<p>Im only getting 1s out of the NN.</p>
</blockquote>
</aside>
<p>Mathematically, <code>sigmoid()</code> will “saturate” and become exactly <code>1</code><br>
when <code>x</code> becomes <code>inf</code> (and become <code>0</code> when <code>x</code> is <code>-inf</code>).</p>
<p>For floats, <code>sigmoid()</code> becomes exactly <code>1.0</code> somewhere around<br>
<code>x = 17.0</code>.</p>
<p>Check the values of <code>x</code> going into <code>sigmoid()</code>.</p>
<p>How large are your inputs?  Do you only get 1s out even on your<br>
first pass before you start training?</p>
<p>By default, your <code>Linear</code> layers are constructed with “sensible”<br>
random values for their weights.  If your inputs are of order 1,<br>
then before you start training (that is, before the initial values of<br>
the weights have changed) the <code>x</code> going into <code>sigmoid()</code> should<br>
have sensible values, and you shouldn’t be getting all exactly 1s.</p>
<p>If the output of your network starts out not being all 1s, but then<br>
becomes all 1s after training for a while, then your training is<br>
pushing your network to return all 1s (which may or may not<br>
make sense, depending on what you are training your network<br>
to do).</p>
<aside class="quote no-group">
<blockquote>
<p>I also started with using the optim.Adam and the MSELoss function</p>
</blockquote>
</aside>
<p>What are typical <code>batch_Y</code> (<code>target</code>) values?  And what do they<br>
mean?  What are you trying to train your network to do?</p>
<p>Best.</p>
<p>K. Frank</p>

ApeelingPotato: <p>The inputs are acceleration and gyroscope data and timing of that data. The purpose of the NN is a study into the effectiveness of NN to measure stride lengths. I am getting values greater than one now thankfully, I think I was not running the training for long enough. The problem now is that for a test batch of 200 inputs the output remains the same.<br>
Typical values range from 1.2514 to 1.5568. Is there any way to get it so it gives a unique float output for each input in this range?</p>

KFrank: <p>Hi Brian!</p>
<aside class="quote no-group" data-username="ApeelingPotato" data-post="6" data-topic="77250" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f07891/48.png" class="avatar"> ApeelingPotato:</div>
<blockquote>
<p>The inputs are acceleration and gyroscope data and timing of that data. The purpose of the NN is a study into the effectiveness of NN to measure stride lengths.</p>
</blockquote>
</aside>
<p>Okay, that sounds reasonable.</p>
<p>But what are the <code>batch_Y</code> (targets)?  Are they known stride lengths?</p>
<aside class="quote no-group">
<blockquote>
<p>I am getting values greater than one now thankfully.</p>
</blockquote>
</aside>
<p>We’re not on the same page here.  You <em>can’t</em>  get a value greater<br>
than one out of a <code>sigmoid()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>The problem now is that for a test batch of 200 inputs the output remains the same.</p>
</blockquote>
</aside>
<p>I don’t really understand what you mean.  Do you mean that you have<br>
a batch size of 200, and for each sample in the batch you get exactly<br>
the same value for the single, scalar output?  (Also, I thought you<br>
said earlier that your batch size was ten.)</p>
<p>Do you mean that the values of your outputs remains exactly the<br>
same (for the same inputs) even as you train your network?</p>
<aside class="quote no-group">
<blockquote>
<p>Typical values range from 1.2514 to 1.5568. Is there any way to get it so it gives a unique float output for each input in this range?</p>
</blockquote>
</aside>
<p>Typical values of what?  Are you referring to input values?  Output<br>
values?</p>
<p>What do you mean by a “unique float output?”  A generic floating-point<br>
number is pretty close to unique?  (There are approximately 2^32<br>
different floating-point numbers, so any two generic values will almost<br>
certainly differ.)</p>
<p>Best.</p>
<p>K. Frank</p>

ApeelingPotato: <p>Sorry for the confusion in the last post and thank you for being patient with me.</p>
<blockquote>
<p>But what are the  <code>batch_Y</code>  (targets)? Are they known stride lengths?</p>
</blockquote>
<p>These are known stride lengths (outputs) in the range 1.2514 to 1.5568 that in batches of 10 in a tensor.</p>
<blockquote>
<p>I don’t really understand what you mean. Do you mean that you have a batch size of 200, and for each sample in the batch you get exactly the same value for the single, scalar output?</p>
</blockquote>
<p>For the test batch, I meant I have separate data that I compare my NN with. It has 200 tensor(1, 1040) inputs with their respective outputs.</p>
<p>The issue I have is when I feed an input tensor into the neural network, it feeds out a tensor with the same value single float value for each input. I want the output to be specific with each input. The training and testing code is as follows with the NN class as above</p>
<pre><code class="lang-auto">    val_size = int(len(input_tensor) * test_partition)
    train_inputs = input_tensor[:-val_size]
    train_outputs = output_tensor[:-val_size]
    test_inputs = input_tensor[-val_size:]
    test_outputs = output_tensor[-val_size:]

        for epoch in range(EPOCHS):
        for i in tqdm(range(0, (len(train_inputs)), BATCH_SIZE)):
            batch_X = train_inputs[i:i+BATCH_SIZE]
            batch_Y = train_outputs[i:i+BATCH_SIZE]

            net.zero_grad()
            outputs = net(batch_X)
            loss = loss_function(outputs, batch_Y)
            loss.backward()

            if epoch &gt; learning_rate_cutoff:  # this acts as a decaying learning rate for our system
                learning_rate = 0.0001
            optimizer.step()
    print(f"\nEpoch: {epoch}. Loss: {loss}")
    
    print("testing")
    correct = 0
    total = 0
    with torch.no_grad():
        for i in tqdm(range(len(test_inputs))):
            predicted_class = net(test_inputs[i].view(-1, 1040))
            real_class = test_outputs[i]
            print(predicted_class, real_class)

            if predicted_class == real_class:
                correct += 1
            total += 1
    print("\nAccuracy: ", round(correct / total, 3))
</code></pre>
<p>EPOCHS is 700<br>
BATCH_SIZE is 10<br>
test_partition is 0.1 with len(input_tensor) being 2000</p>

KFrank: <p>Hello Brian!</p>
<aside class="quote no-group" data-username="ApeelingPotato" data-post="8" data-topic="77250" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f07891/48.png" class="avatar"> ApeelingPotato:</div>
<blockquote>
<p>The issue I have is when I feed an input tensor into the neural network, it feeds out a tensor with the same value single float value for each input. I want the output to be specific with each input.</p>
</blockquote>
</aside>
<p>Do I understand correctly?</p>
<p>You feed <code>input1</code> – a single input tensor consisting of 1040 floats — into<br>
your network, and you get <code>outputA</code> – a single float.  You then feed in a<br>
different input tensor, <code>input2</code>, but get the same output, <code>outputA</code>.</p>
<p>This seems very odd.</p>
<p>Could you post a simple, complete, runnable script, together with its<br>
output, that demonstrates this issue?</p>
<p>I have in mind something like:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import ...

print (torch.__version__)

input1 = &lt;some explicit numerical input tensor&gt;
input2 = &lt;another explicit numerical input tensor&gt;
input3 = &lt;yet another&gt;

class Net(nn.Module):
    &lt;your specific network that displays this issue&gt;

net = Net()

output1 = net (input1)
print ('input1 = ...')
print (input1)
print ('output1 = ...')
print (output1)

output2 = net (input2)
print ('input2 = ...')
print (input2)
print ('output2 = ...')
print (output2)

output3 = net (input3)
print ('input3 = ...')
print (input3)
print ('output3 = ...')
print (output3)

print ('output2 - output1 =', output2 - output1)
print ('output3 - output1 =', output3 - output1)
</code></pre>
<p>The idea is that others should be able to run the script you post, and<br>
by doing so, reproduce the output you post.</p>
<p>Best.</p>
<p>K. Frank</p>

ApeelingPotato: <p>The error is on my end, I was running the NN as a classification network as opposed to a regression style. I followed an example for predicting house prices on this <a href="https://docs.microsoft.com/en-us/archive/msdn-magazine/2019/march/test-run-neural-regression-using-pytorch" rel="nofollow noopener">link</a> and have been able to find what I was looking for. Thank you so much for your time.</p>