How to implement a neural network with 2-d tensor as input and output?

username: jarvico
cooked: <p>Hi,</p>
<p>I am trying to train a simple NN which takes 2-d tensor as input data, and outputs a 2-d tensor. Specifically, I would like to have an input and output of shape 16x2.</p>
<p>Below is the model I used.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    
    def __init__(self): 
        super(MyModel, self).__init__()

        self.fc1 = nn.Linear(16, 256)
        self.fc2 = nn.Linear(256, 512)
        self.fc3 = nn.Linear(512, 256)
        self.fc4 = nn.Linear(256, 16)
        
    def forward(self, x):
        
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.sigmoid(self.fc4(x))
        
        return x
</code></pre>
<p>With a batch size of 1, I get this error message:</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x2 and 16x256)
</code></pre>
<p>And, with a batch size of 128 (my preferred option is this, as I need to train my model), I get this error message:</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (2048x2 and 16x256)
</code></pre>
<p>Would you please help me how to use a batch of tensors with dimensions 16x2 as input and having a batch of tensors with dimensions 16x2 as output.</p>
<p>Obviously, if batch size is 128, input and output dimensions should be 128x16x2.</p>
<p>Best wishes…</p>

username: srishti-git1110
cooked: <p>Hi,<br>
You are facing this error because your first linear layer expects an input vector of size 16 while you are using as input a tensor of dimension batch_size * 16 * 2.</p>
<p>Either change the dimensions of your input tensor or have a look at other layers like convolutional layers in case you are working with images.</p>

username: jarvico
cooked: <p>Hi,</p>
<p>Thanks for your prompt response.</p>
<p>The thing is I would like to have the outputs of all layer’s activations as 2d tensor (ignoring the batch size for now).</p>
<p>For example, when I fed the input (16x2), I want the dimensions of first layer’s output as 256x2, second layer’s output as 512x2, third layer’s output as 256x2 and output layer output as 16x2.</p>
<p>I hope it is clear now.</p>
<p>I guess it is not possile with nn.linear , right ? any other possibility ?</p>

username: srishti-git1110
cooked: <p>I am not sure what your use case in terms of the type of data and modelling problem is.<br>
But, you could try <a href="https://pytorch.org/docs/stable/generated/torch.reshape.html#torch.reshape" rel="noopener nofollow ugc">reshape/view</a> to change your 1 dimensional tensor to 2 dimensional.</p>
<p>Feel free to post further queries.</p>

username: jarvico
cooked: <p>Hi,</p>
<p>I already use an input as 2d with shape (16x2) when feeding to the model. (ignoring batch size)</p>
<p>when I feed my input to the model above, I got below error.<br>
dimensions of my input are batch_size x (16x2). batch size is 128.</p>
<pre><code class="lang-auto">Input In [5], in MyModel.forward(self, x)
     14 def forward(self, x):
     16     print("shape of x initially is ", x.shape) #shape of x initially is  torch.Size([128, 16, 2])
---&gt; 18     x = F.relu(self.fc1(x))
     19     x = F.relu(self.fc2(x))
     20     x = F.relu(self.fc3(x))

File ~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\modules\module.py:1102, in Module._call_impl(self, *input, **kwargs)
   1098 # If we don't have any hooks, we want to skip the rest of the logic in
   1099 # this function, and just call forward.
   1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102     return forward_call(*input, **kwargs)
   1103 # Do not call functions when jit is used
   1104 full_backward_hooks, non_full_backward_hooks = [], []

File ~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\modules\linear.py:103, in Linear.forward(self, input)
    102 def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 103     return F.linear(input, self.weight, self.bias)

File ~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\functional.py:1848, in linear(input, weight, bias)
   1846 if has_torch_function_variadic(input, weight, bias):
   1847     return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-&gt; 1848 return torch._C._nn.linear(input, weight, bias)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2048x2 and 16x256)

</code></pre>
<p>if, for example, my batch size is 128 as in my case, my requirement is below:</p>
<p>input shape: 128 x 16 x 2<br>
1st hidden layer output shape: 128 x 256 x 2<br>
2nd hidden layer output shape: 128 x 512 x 2<br>
3rd hidden layer output shape: 128 x 256 x 2<br>
output layer shape: 128 x 16 x 2</p>
<p>I wonder how to design and implement such a NN.</p>
<p>Best wishes,</p>

username: srishti-git1110
cooked: <p>Hi,</p>
<aside class="quote no-group" data-username="jarvico" data-post="5" data-topic="166787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>when I feed my input to the model above, I got below error.</p>
</blockquote>
</aside>
<p>This is expected as</p>
<aside class="quote no-group" data-username="srishti-git1110" data-post="2" data-topic="166787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p>You are facing this error because your first linear layer expects an input vector of size 16 while you are using as input a tensor of dimension batch_size * 16 * 2.</p>
</blockquote>
</aside>
<p>Please see if this helps:</p>
<pre data-code-wrap="python"><code class="lang-python">class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.fc1 = nn.Linear(32, 512)
    self.fc2 = nn.Linear(512, 1024)
    self.fc3 = nn.Linear(1024, 512)
    self.fc4 = nn.Linear(512, 32)

  def forward(self, x):
    x = x.reshape(-1, 32) # reshaping the input tensor to (batch_size * 32)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = F.sigmoid(self.fc4(x))
    x = x.reshape(-1, 16, 2)
    return x

inp = torch.rand(128, 16, 2)     
net = Model()
out = net(inp)
print(out.shape) # torch.Size([128, 16, 2])
</code></pre>
<aside class="quote no-group" data-username="jarvico" data-post="5" data-topic="166787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>1st hidden layer output shape: 128 x 256 x 2<br>
2nd hidden layer output shape: 128 x 512 x 2<br>
3rd hidden layer output shape: 128 x 256 x 2</p>
</blockquote>
</aside>
<p>You could use similar reshape calls to change the dimensions of hidden layer tensors if you require to use these intermediate outputs in any way.</p>
<p>Best,<br>
S</p>

username: jarvico
cooked: <p>Thanks for your help.</p>
<p>But the thing 2-d data is my strict requirement. I have to solve this issue without reshaping to 1d tensor first and reshaping to 2d tensor at the end.</p>
<p>Again, if, for example, my batch size is 128 as in my case, my requirement is below:</p>
<p>input shape: 128 x 16 x 2<br>
1st hidden layer output shape: 128 x 256 x 2<br>
2nd hidden layer output shape: 128 x 512 x 2<br>
3rd hidden layer output shape: 128 x 256 x 2<br>
output layer shape: 128 x 16 x 2</p>
<p>I want my NN to handle this 2d data throughout the whole feedforward and backforward passes.</p>

username: srishti-git1110
cooked: <aside class="quote no-group" data-username="jarvico" data-post="7" data-topic="166787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>But the thing 2-d data is my strict requirement.</p>
</blockquote>
</aside>
<p>In that case, too, I think you will have to permute your tensor dimensions as for the <code>nn.Linear</code> layer to be able to process multi-dimensional inputs, the last dimension of the input tensor must match the <code>in_features</code> and the output tensor is then the same shape as the input tensor but for the last dimension which is equal to <code>out_features</code>.</p>
<p>So, input 128 x 16 x 2 cannot be mapped to 128 x 256 x 2.<br>
Although mapping 128 x 2 x 16 to 128 x 2 x 256 is certainly possible.</p>
<p>If it satisfies your use case, you could use <code>torch.permute</code> to re-arrange your tensors like so:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
inp = torch.rand(128, 16, 2)
inp = torch.permute(inp, (0, 2, 1))
inp.shape # torch.Size([128, 2, 16])
</code></pre>
<p>and further redefine your network. See:</p>
<pre data-code-wrap="python"><code class="lang-python">fc1 = torch.nn.Linear(16, 256)
out = fc1(inp)
print(out.shape) # torch.Size([128, 2, 256])
</code></pre>

username: vdw
cooked: <aside class="quote no-group" data-username="jarvico" data-post="7" data-topic="166787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>But the thing 2-d data is my strict requirement. I have to solve this issue without reshaping to 1d tensor first and reshaping to 2d tensor at the end</p>
</blockquote>
</aside>
<p>Just out of curiosity, why?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/jarvico">@jarvico</a>,</p>
<p>Following on from <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> answer, you can get the desired behaviour by simply transposing and exploit broadcasting within a <code>nn.Linear</code> layer.</p>
<pre><code class="lang-auto">import torch
from torch import nn, Tensor


class model(nn.Module):

  def __init__(self):
    super(model ,self).__init__()
    layers = []
    hid_in, hid_out = [16, 256, 512, 256], [256, 512, 256, 16]
    for hin, hout in zip(hid_in, hid_out):
      layers.append(nn.Linear(hin, hout))
    self.layers = nn.ModuleList(layers)

  def forward(self, x: Tensor) -&gt; Tensor:
    for i, layer in enumerate(self.layers):
      x = layer(x.transpose(-2,-1)).transpose(-2,-1) #transpose to exploit broadcasting over 2d-input
      print(f"Layer {i}, size: {x.shape}")
    return x

x=torch.randn(128,16,2)

net = model()
net(x)
</code></pre>
<p>This returns</p>
<pre><code class="lang-auto">Layer 0, size: torch.Size([128, 256, 2]) #hidden 1
Layer 1, size: torch.Size([128, 512, 2]) #hidden 2
Layer 2, size: torch.Size([128, 256, 2]) #hidden 3
Layer 3, size: torch.Size([128, 16, 2]) #output 
</code></pre>

username: jarvico
cooked: <p>Thank you very much, this works!</p>