How to build a view layer in Pytorch for Sequential Models?

pinocchio: <p>How to build a view layer in Pytorch for Sequential Models?</p>
<p>Is this ok:</p>
<pre><code class="lang-auto">class View(nn.Module):
    def forward(self, input, shape):
        return input.view(*shape)
</code></pre>
<p>I tried it based on the flatten layer but I couldn’t even make the flatten layer work:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

## Q: why does the flatten layer work?
class Flatten(nn.Module):
    def forward(self, input):
        print(input.size())
        out = input.view(input.size(0),-1)
        print(out.size())
        return out

class View(nn.Module):
    def forward(self, input, shape):
        return input.view(*shape)

def main():
    x = torch.arange(0,6).view(3,2)
    print(x)
    print(x.size())
    flatten = Flatten()
    flt_x = flatten(x)
    print(flt_x)
    print(flt_x.size())

if __name__ == '__main__':
    main()
</code></pre>
<p>look at the weird output doesn’t chage size:</p>
<pre><code class="lang-auto">tensor([[0, 1],
        [2, 3],
        [4, 5]])
torch.Size([3, 2])
torch.Size([3, 2])
torch.Size([3, 2])
tensor([[0, 1],
        [2, 3],
        [4, 5]])
torch.Size([3, 2])
</code></pre>

albanD: <p>Hi,</p>
<p>This seems to work no? You keep the first dimension and collapse all the others. But your Tensor had only 2 dimensions to begin with.</p>
<p>By the way for use within a Sequential, you can define a custom <code>__init__()</code> function on your View Module that will take the shape as input.</p>

pinocchio: <p>ok perhaps this clarifies my confusion:</p>
<pre><code class="lang-auto">class Flatten(nn.Module):
    def forward(self, input):
        '''
        Note that input.size(0) is usually the batch size.
        So what it does is that given any input with input.size(0) # of batches,
        will flatten to be 1 * nb_elements.
        '''
        batch_size = input.size(0)
        out = input.view(batch_size,-1)
        return out # (batch_size, *size)

class View(nn.Module):
    def forward(self, input, shape):
        '''
        TODO: the first dimension is the data batch_size
        so we need to decide how the input shape should be like
        '''
        return input.view(*shape)
</code></pre>

pinocchio: <p>oh I see…ur saying I can’t pass “shape” as input…</p>

pinocchio: <p>So this is correct:</p>
<pre><code class="lang-auto">
class View(nn.Module):

    def __init__(self, shape):
        self.shape = shape

    def forward(self, input):
        '''
        TODO: the first dimension is the data batch_size
        so we need to decide how the input shape should be like
        '''
        return input.view(*self.shape)
</code></pre>

albanD: <p>If you want to use the View in a sequential yes. You have to do this. Because the Sequential only passes the output of the previous layer.</p>
<p>For your Flatten layer, it seem to work fine no?</p>
<pre><code class="lang-auto">import torch
from torch import nn

class Flatten(nn.Module):
    def forward(self, input):
        '''
        Note that input.size(0) is usually the batch size.
        So what it does is that given any input with input.size(0) # of batches,
        will flatten to be 1 * nb_elements.
        '''
        batch_size = input.size(0)
        out = input.view(batch_size,-1)
        return out # (batch_size, *size)

print("2D input")
foo = torch.rand(10, 20)
print("Input size:")
print(foo.size())
bar = Flatten()(foo)
print("Output size:")
print(bar.size())

print("3D input")
foo = torch.rand(10, 20, 30)
print("Input size:")
print(foo.size())
bar = Flatten()(foo)
print("Output size:")
print(bar.size())

print("8D input")
foo = torch.rand(10, 2, 3, 4, 5, 6, 7, 8)
print("Input size:")
print(foo.size())
bar = Flatten()(foo)
print("Output size:")
print(bar.size())
</code></pre>

pinocchio: <p>correction:</p>
<pre><code class="lang-auto">class View(nn.Module):
    def __init__(self, shape):
        super().__init__()
        self.shape = shape

    def forward(self, input):
        '''
        Reshapes the input according to the shape saved in the view data structure.
        '''
        batch_size = input.size(0)
        shape = (batch_size, *self.shape)
        out = input.view(shape)
        return out
</code></pre>

pinocchio: <p>I made my layer but I get a weird error:</p>
<pre><code class="lang-auto">  File "/Users/pinocchio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 488, in __call__
    for hook in self._forward_pre_hooks.values():
  File "/Users/pinocchio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 539, in __getattr__
    type(self).__name__, name))
AttributeError: 'View' object has no attribute '_forward_pre_hooks'
</code></pre>
<p>you know why?</p>

albanD: <p>Make sure to properly call the parent <code>__init__</code> function when creating your own <code>nn.Module()</code>.<br>
Also make sure that you don’t have any other class/function called <code>View</code> that could conflict with your new one.</p>

pinocchio: <p>hmmm I will try debugging a little longer. Meanwhile let me paste the code I ran (that had an error) for reference:</p>
<pre><code class="lang-auto">        ##
        batch_size = 1
        CHW = (3, 32, 32)
        out = torch.randn(batch_size,*CHW)
        print(f'out.size()')
        ##
        conv2d_shape = (-1, 8, 8)
        view = View(shape=(batch_size,*conv2d_shape))
        ##
        out = view(out)
        print(f'out.size()')
</code></pre>

pinocchio: <aside class="quote no-group quote-modified" data-username="pinocchio" data-post="7" data-topic="53958">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>super().<strong>init</strong>()</p>
</blockquote>
</aside>
<p>Darn it! I was using a wrong version of my View layer. Ooops! Fixed.</p>

pinocchio: <pre><code class="lang-auto">class View(nn.Module):
    def __init__(self, shape):
        super().__init__()
        self.shape = shape

    def __repr__(self):
        return f'View{self.shape}'

    def forward(self, input):
        '''
        Reshapes the input according to the shape saved in the view data structure.
        '''
        batch_size = input.size(0)
        shape = (batch_size, *self.shape)
        out = input.view(shape)
        return out
</code></pre>