Dimension of the input of a Neural Network built with sequential

username: miraboreasu
cooked: <p>Hello,</p>
<p>I have built a NN like this</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Sequential(
            nn.Linear(1, 10),
            nn.ReLU(),
            nn.Linear(10, 1),
            nn.ReLU(),
        )

    def forward(self, x):
        out = self.fc(x)
        return out
</code></pre>
<p>to call the forward, I do</p>
<pre><code class="lang-auto">x=torch.tensor(np.array([1, 2, 3]), dtype=torch.float)
net=Net()
</code></pre>
<p>since the neuron dimension in the module is 1x10 (please correct me if I am wrong), I use <code>.view</code> to reshape my data, now I have seen two examples,</p>
<p>one is</p>
<pre><code class="lang-auto">outputs=net(x.view(-1, 1))
</code></pre>
<p>another is</p>
<pre><code class="lang-auto">outputs=net(x.view(-1, 1, 1))
</code></pre>
<p>They seem to work fine</p>
<p>But I am confused, why they work both, can anyone help me?</p>

username: Matias_Vasquez
cooked: <p>When using <code>nn.Linear</code>, the <strong>last</strong> dimension of your input has to match the <strong>first</strong> dimension of your linear layer.</p>
<p>This means that it does not matter how many dimensions there are before, and how big/small they are, the last one has to be equal to <code>in_features</code> from your <code>nn.Linear</code>.</p>
<p>In your example <code>in_features=1</code>. This means that any tensor with <code>1</code> in the last dimension will work.</p>
<pre><code class="lang-python"># Example
in_features = 1
out_features = 10

linear = torch.nn.Linear(in_features, out_features)

inp = torch.rand(1, 2, 3, 4, in_features)
output = linear(inp)

print(output.shape)
# Output
# torch.Size([1, 2, 3, 4, 10])
</code></pre>
<p>You can play with this. You can change how many dimensions <code>inp</code> has. You can also change how big <code>in_features</code> is. The important thing is that it is the same value in the LAST dim for <code>inp</code> and FIRST for <code>nn.Linear</code>.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
<p>(If you want to know why, it is because it is performing a matrix multiplication, therefore the inner dims must match. There is more information on the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">docs</a>)</p>

username: miraboreasu
cooked: <p>Thank you, I found that influences the training results. If I use <code>outputs=net(x.view(-1, 1))</code>, the result is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84877f7afbb52bbf8aaa2ac40fd3ad6b98c76580.png" alt="cf246ac25c73677ff454f3d1bf77685" data-base62-sha1="iUpkpitQBTNEm5t187gx3yzRlja" width="542" height="395"><br>
If I use <code>outputs=net(x.view(-1, 1, 1 ))</code>, the result is<br>
(<a href="https://ibb.co/XjYfsbm" class="inline-onebox" rel="noopener nofollow ugc">20220516132715 hosted at ImgBB — ImgBB</a>)</p>
<p>Sorry I can only upload one picture, so I put it online</p>
<p>Do you know why?</p>

username: Matias_Vasquez
cooked: <p>We can take a look at a little example</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.rand(3, 2)
</code></pre>
<p>If we print <code>a</code> we get something like this</p>
<pre data-code-wrap="python"><code class="lang-python">print(a)
# tensor([[0.9699, 0.6693],
#         [0.1688, 0.0404],
#         [0.7409, 0.6294]])
</code></pre>
<p>If we print it like this, it will just flatten the tensor into a one dimensional vector.</p>
<pre data-code-wrap="python"><code class="lang-python">print(a.view(-1))
# tensor([0.9699, 0.6693, 0.1688, 0.0404, 0.7409, 0.6294])
</code></pre>
<p>If we now use the first of your options, we see that we gained a dimension. Now, each value is inside its own vector.</p>
<pre data-code-wrap="python"><code class="lang-python">print(a.view(-1, 1))
# tensor([[0.9699],
#         [0.6693],
#         [0.1688],
#         [0.0404],
#         [0.7409],
#         [0.6294]])
</code></pre>
<p>If we now do your second option we gain another dimension. The values are further isolated.</p>
<pre data-code-wrap="python"><code class="lang-python">print(a.view(-1, 1, 1))
#tensor([[[0.9699]],
#
#        [[0.6693]],
#
#        [[0.1688]],
#
#        [[0.0404]],
#
#        [[0.7409]],
#
#        [[0.6294]]])
</code></pre>
<p>However, the values are always the same.</p>
<p>You are then feeding these values to a <code>nn.Linear</code> layer that might look like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/041629f166b942dfb18318a729739c7d937d67ee.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/041629f166b942dfb18318a729739c7d937d67ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041629f166b942dfb18318a729739c7d937d67ee_2_345x224.jpeg" alt="image" data-base62-sha1="A9ox1S6o8Yqv2o2EBSH46qxhq6" width="345" height="224" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041629f166b942dfb18318a729739c7d937d67ee_2_345x224.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041629f166b942dfb18318a729739c7d937d67ee_2_517x336.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041629f166b942dfb18318a729739c7d937d67ee_2_690x448.jpeg 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1248×812 45.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But as we mentioned, the input values remain the same. We only added more “isolation” to the values, but they are both of the size <code>1</code>. So, the same inputs are going to the same network.</p>
<p>So, my guess for this particular case is that this should not play that much of a role. It would be more significant if you played with the input size of your linear layer (which should match the last dimension of your input).</p>
<p>In order to be sure that one <code>view</code> performs better than the other you could take two different approaches.</p>
<ul>
<li>
<p><strong>Deterministic approach</strong>: You can make sure that all of the random variables have the same seed when trying both approaches to see if there is really a difference in training.<br>
<a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Here</a> you can read more about it. For this simple approach I think it would be enough to set <code>torch.manual_seed(0)</code> before creating your <code>net</code>. This way the weights of your linear layer are the same for both cases.</p>
</li>
<li>
<p><strong>Many repetitions:</strong> You could also just try a bunch of times with both <code>views</code> and see if you get significantly better results with one than with the other.</p>
</li>
</ul>
<p>If the results do show that one <code>view</code> is better than the other, then I do not know what might be happening.</p>
<p>Please let me know what you find out.</p>