Add module to nn.Sequential

RFTEXAS: <p>Hello I’m quite new to PyTorch. I’m trying to implement a Neural Net originally designed with Keras.</p>
<p>In particular, I have an issue with the Add module of Keras:</p>
<pre><code class="lang-auto"># C is also a Conv1D layer
C12 = Conv1D(filters=32, kernel_size=5, strides=1, padding='same')(A11)
S11 = Add()([C12, C])
</code></pre>
<p>I don’t find any equivalent in PyTorch. In particular, I’d like to implement it using the nn.Sequential design.</p>
<p>Any help is welcome.</p>
<p>Many thanks in advance</p>
<p>Best,<br>
PAB</p>

G.M: <p>Try <code>nn.Sequential(C12, C)</code> Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

RFTEXAS: <p>I thought that nn.Sequential would pass my input sequentially into C12 and C. I think that S11 adds the ouput of C12 and C, doesn’t it ?</p>
<p>torch.add might be the solution but I don’t know how it can fit into nn.Sequential since I don’t specify which layers to add.</p>

G.M: <p>U r correct about <code>nn.Sequential</code>. I don’t get what r u trying to achieve, so can u explain more about what r u trying to do since I’m not familiar with Keras?</p>

RFTEXAS: <p>I am trying to implement a model from an Arxiv paper. The author used Keras to implement it. I thought that a relevant way to implement it using PyTorch is by using nn.Sequential . Here is the architecture of the model.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d154996c0fb1ad8210959b4c030ae84810e5937.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d154996c0fb1ad8210959b4c030ae84810e5937" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d154996c0fb1ad8210959b4c030ae84810e5937_2_222x500.png" alt="Capture" data-base62-sha1="fyZErZKj0OvKx480Cr5ijQUDH03" width="222" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d154996c0fb1ad8210959b4c030ae84810e5937_2_222x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d154996c0fb1ad8210959b4c030ae84810e5937.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d154996c0fb1ad8210959b4c030ae84810e5937.png 2x" data-dominant-color="F0F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">282×634 27.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I understand the concept of nn.Sequential but I don’t know how to include torch.add() in a nn.Sequential block.</p>

G.M: <p>I don’t think there is something in Pytorch that does it; u probably need to split the Sequential and add them manually.</p>

Eta_C: <p>If you want to insert some modules dynamically, try <code>nn.ModuleList</code> instead of <code>nn.Sequential</code>. Some operators such as <code>nn.ModuleList.append</code>, <code>nn.ModuleList.extend</code>, <code>nn.ModuleList.insert</code> may helpful.</p>
<p>In my understanding, do you want to dynamically choose whether to use skip connection(ADD)? I have an idea</p>
<pre><code class="lang-python">self.door = 1  # when you define your model
S11 = C12 + C * self.door # in forward function

net.door = 0 # if you want to drop the skip connection
net.door = 1 # if you want to keep the skip connection
</code></pre>

fodierna: <p>I am trying to implement the same NN. This is the way I build the residual blocks:</p>
<p>class ResidualBlock(nn.Module):</p>
<pre><code>def __init__(self, in_channels, out_channels, kernel_size, stride):
    super(ResidualBlock, self).__init__()

    self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size)
    self.conv2 = nn.Conv1d(in_channels, out_channels, kernel_size)
    self.max_pool = nn.MaxPool1d(kernel_size, stride)


def forward(self, x):
    residual = x

    out = F.relu(self.conv1(x))
    out = F.relu(self.conv2(out))

    out += residual

    out = self.max_pool(F.relu(out))

    return out
</code></pre>
<p>However, I am having problems because the residual’s shape doesn’t match the output’s shape of the second conv layer. Any idea to solve the problem?</p>

ptrblck: <p>You could add <code>padding</code> to the conv layers and make sure the spatial size isn’t decreased or alternatively, you could downsample the <code>residual</code> as shown in the <a href="https://github.com/pytorch/vision/blob/7b60f4db9707d7afdbb87fd4e8ef6906ca014720/torchvision/models/resnet.py#L66-L67">resnet implementation</a>.</p>

fodierna: <p>Thank you! It works.</p>
<p>Some advices for those who are trying to reproduce <a href="https://arxiv.org/abs/1805.00794:" rel="nofollow noopener">https://arxiv.org/abs/1805.00794:</a></p>
<ul>
<li>be careful with the lr_scheduler, it must be reset after each training epoch. The steps must be performed each 10k training samples.</li>
<li>the softmax after the second fully connected layer is not required.</li>
</ul>
<p>This is the code to build the NN as described in the paper. Hope this helps!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class ECG_CNN(nn.Module):

    def __init__(self, num_classes=5):

        super(ECG_CNN, self).__init__()

        self.conv1 = nn.Conv1d(1, 32, kernel_size=5)
        
        self.res1 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
        self.res2 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
        self.res3 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
        self.res4 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
        self.res5 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)

        self.fc1 = nn.Linear(32*2, 32)
        self.fc2 = nn.Linear(32, num_classes)
    

    def forward(self, x):
        out = self.conv1(x)

        out = self.res1(out)
        out = self.res2(out)
        out = self.res3(out)
        out = self.res4(out)
        out = self.res5(out)

        out = torch.flatten(out, 1)
        out = F.relu(self.fc1(out))
        out = self.fc2(out)

        return out

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride, padding):
        super(ResidualBlock, self).__init__()

        self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size, padding=padding)
        self.conv2 = nn.Conv1d(in_channels, out_channels, kernel_size, padding=padding)
        self.max_pool = nn.MaxPool1d(kernel_size, stride=stride)
    

    def forward(self, x):
        residual = x

        out = F.relu(self.conv1(x))
        out = self.conv2(out)

        out += residual

        out = self.max_pool(F.relu(out))

        return out
</code></pre>