4D tensor equivalent neural network layer in PyTorch

username: ahmadreza9
cooked: <p>Hello,<br>
How can I define a layer like below code in pytorch?</p>
<pre><code class="lang-auto">InputLayer(
        shape=(None, 1, input_height, input_width),
    )
</code></pre>
<p>(The input is a 4 Dimensional tensor.)</p>

username: SiQube
cooked: <p>Hey ahmad,</p>
<p>there is no need to specify an input layer, you can directly start using convolutional or linear layers in pytorch via torch.nn.Conv or torch.nn.Linear.</p>
<p>If you have any more questions, just post them.</p>

username: ahmadreza9
cooked: <p>@ <a href="https://discuss.pytorch.org/u/SiQube">SiQube</a> I have below neural network in Theano library, and want to find its equivalent neural network in PyTorch.</p>
<pre><code class="lang-auto">l_in = lasagne.layers.InputLayer(
        shape=(None, 1, input_height, input_width),
    )

    l_hid = lasagne.layers.DenseLayer(
        l_in,
        num_units=20,
        # nonlinearity=lasagne.nonlinearities.tanh,
        nonlinearity=lasagne.nonlinearities.rectify,
        # W=lasagne.init.Normal(.0201),
        W=lasagne.init.Normal(.01),
        b=lasagne.init.Constant(0)
    )

    l_out = lasagne.layers.DenseLayer(
        l_hid,
        num_units=output_length,
        nonlinearity=lasagne.nonlinearities.softmax,
        # W=lasagne.init.Normal(.0001),
        W=lasagne.init.Normal(.01),
        b=lasagne.init.Constant(0)
    )
</code></pre>
<p>I just want to know how covet l_in layer to PyTorch layer and pass it to L-hid.<br>
I wrote below code which is not completed but it could clarify what I need.</p>
<pre><code class="lang-auto"> def __init__(self, input_height, input_width, output_length):
        super(Net, self).__init__()

        **self.fc1 = nn.Linear(input_height * input_width, 20)**  (this  layer must be change and become sutable for 4D tensor)
        self.fc1.weight.data.fill_(torch.nn.init.normal_(.01))
        self.fc1.bias.data.fill_(torch.nn.init.constant(0))
        self.fc2 = nn.Linear(20, output_length)
        self.fc2.weight.data.fill_(torch.nn.init.normal_(.01))
        self.fc2.bias.data.fill_(torch.nn.init.constant(0))

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.softmax(self.fc2(x))
        return x
</code></pre>

username: thomasjo
cooked: <p>The code for your layers looks fine. There are multiple ways in PyTorch for flattening a multi-dimensional input. In your case, one option is to change your <code>forward</code> method to something like</p>
<pre><code class="lang-python">def forward(self, x):
    x = torch.flatten(x, start_dim=1)  # Assumes input 'x' is batched
    x = F.relu(self.fc1(x))
    x = F.softmax(self.fc2(x))
    return x
</code></pre>
<p>Note that you could also combine the flatten instruction with the first line in your example;</p>
<pre><code class="lang-python">def forward(self, x):
    x = F.relu(self.fc1(x.flatten(start_dim=1))  # Assumes input 'x' is batched
    x = F.softmax(self.fc2(x))
    return x
</code></pre>
<p>Personally, I prefer the first alternative since I find it more readable.</p>
<p><strong>UPDATE:</strong> You could also solve this by using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Flatten" rel="nofollow noopener"><code>torch.nn.Flatten</code></a>.</p>

username: SiQube
cooked: <p>Oh, that you can in the forward pass, you reshape your tensor via</p>
<pre><code class="lang-auto">def forward(self, x):
    x = x.view(batch_size, -1)
    x = F.relu(self.fc(x))
</code></pre>
<p>…</p>

username: ahmadreza9
cooked: <p>Thank you for your attention. Now I don’t want to flatting my input.  My input is (10, 1, 20, 224). I want to pass this tensor to l_in but I don’t know how pass it to first layer of my network and how pass result of this layer to fc2.<br>
Simply I want equivalent of <code>l_in = lasagne.layers.InputLayer(         shape=(None, 1, input_height, input_width),     )</code> in constructing my neural network with (10, 1, 20, 224) tensor.</p>

username: thomasjo
cooked: <aside class="quote no-group quote-modified" data-username="ahmadreza9" data-post="6" data-topic="85360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahmadreza9/48/25076_2.png" class="avatar"> ahmadreza9:</div>
<blockquote>
<p>Now I don’t want to flatting my input. My input is (10, 1, 20, 224). I want to pass this tensor to l_in but I don’t know pass it to first layer of my network and how pass result of this layer to fc2.</p>
</blockquote>
</aside>
<p>If you look at the documentation for <a href="https://lasagne.readthedocs.io/en/latest/modules/layers/dense.html#module-lasagne.layers.dense" rel="noopener nofollow ugc"><code>lasagna.layers.DenseLayer</code></a>, you will see that they implictly perform a flattening operation for you whenever the input has more than two axes. In other words, you have been using flattened inputs in Theano, but it was “automagically” performed <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

username: ahmadreza9
cooked: <p>So <a href="https://discuss.pytorch.org/u/thomasjo">thomasjo</a></p>
<pre><code class="lang-auto">self.fc1 = nn.Linear(input_height * input_width, 20)
</code></pre>
<p>must be change to</p>
<pre><code class="lang-auto">self.fc1 = nn.Linear(input, 20)
</code></pre>
<p>which input is (10, 1, 20, 224) tensor and will be flatted by</p>
<pre><code class="lang-auto">x = torch.flatten(x, start_dim=1)  # Assumes input 'x' is batched
</code></pre>
<p>Is it right?</p>

username: thomasjo
cooked: <p>I do not remember the details of Theano’s memory layout, but I am assuming it uses the <em>NCHW</em> format, which means your input dimensions <code>(10, 1, 20, 224)</code> corresponds to</p>
<ul>
<li>batch size of 10,</li>
<li>channel depth of 1,</li>
<li>image height of 20 pixels,</li>
<li>image width of 224 pixels.</li>
</ul>
<p><em>(The image height of 20 pixels does seem a bit odd to me – 224 would be more typical – but I am assuming it is correct. Change the examples below if it is incorrect.)</em></p>
<p>Under these assumptions, you do not need to change your declaration of <code>fc1</code>. That layer expects an input tensor with dimensions <code>[None, 20 * 224]</code> and will output a tensor with dimensions <code>[None, 20]</code>.</p>
<p>The flattening operation will ensure that an input tensor with dimensions <code>[None, 1, 224, 20]</code> is transformed into a tensor with dimensions <code>[None, 1 * 20 * 224]</code>, which is exactly the same as layer <code>fc1</code> expects.</p>
<p>Note that if you switch from images with 1 channel to e.g. 3 channels (RGB), you will need to update your definition of <code>fc1</code> accordingly, but you will not need to change the <code>torch.flatten</code> operation.</p>

username: ahmadreza9
cooked: <p>Sir, <a href="https://discuss.pytorch.org/u/thomasjo">thomasjo</a><br>
If I want to use conventional layer to transfer (10, 1, 20, 244) tensor to 11 output like blow table. How should I change my neural network?<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57d02720e8a44e53e955a5288a39a0e2feb41c14.jpeg" alt="IMG_20200614_161158" data-base62-sha1="cwPvkxM5VsGWmWIoCaOw5d72zqY" width="540" height="264"></p>

username: thomasjo
cooked: <p>This is sort of tangential to your original question, but something like this should work for defining the layers/modules</p>
<pre><code class="lang-python">def __init__(self):
    super().__init__()

    self.conv1 = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=(5, 5), stride=1)
    self.pool1 = nn.AvgPool2d(kernel_size=(2, 2), stride=2)

    self.conv2 = nn.Conv2d(in_channels=8, out_channels=16, kernel_size=(5, 5), stride=1)
    self.pool2 = nn.AvgPool2d(kernel_size=(2, 2), stride=2)

    self.fc1 = nn.Linear(in_features=16 * 5 * 56, out_features=72)
    self.fc2 = nn.Linear(in_features=72, out_features=11)
</code></pre>
<p>Then, in your <code>forward</code> method, you could do something like</p>
<pre><code class="lang-python">def forward(self, x):
    x = self.conv1(x)
    x = nn.functional.relu(x, inplace=True)
    x = self.pool1(x)

    x = self.conv2(x)
    x = nn.functional.relu(x, inplace=True)
    x = self.pool2(x)

    x = torch.flatten(x, start_dim=1)
    x = self.fc1(x)
    x = nn.functional.tanh(x)

    x = self.fc2(x)
    x = nn.functional.softmax(x)

    return x
</code></pre>
<p>Please, note that there exist alternative (and arguably better) ways of defining the network and the forward pass. I chose to present the examples this way to better match your table and hopefully be easier to understand. If you want to improve the code, I highly recommend using the <a href="https://pytorch.org/docs/stable/nn.html#sequential" rel="nofollow noopener"><code>nn.Sequential</code></a> container module. This would allow you to greatly simplify the <code>forward</code> method by declaring each “convolutional block” as a single unit, including ReLU and average pooling. I will leave this as an exercise for you to figure out.</p>
<p>Also be aware that I have not tested any of the code, so it is very likely that I have made some mistakes.</p>