A simple extension of nn.Sequential

username: Wesley_Neill
cooked: <p>Hi there!</p>
<p>I’m working through some Udacity courses on PyTorch and decided to go the extra mile to extend the <code>nn.Sequential</code> class. I wanted to automate defining each layer’s activations by just passing a tuple containing the number of nodes in each class.</p>
<p>So normally if I wanted to perform a forward pass with an already initialized <code>nn.Sequential</code> model, I’d simply use</p>
<pre><code class="lang-auto">out = model(x)
# OR
out = model.forward(x)
</code></pre>
<p>Now that I’ve extended the class, I am trying to use</p>
<pre><code class="lang-auto">out = self(x)
# OR
out = self.forward(x)
</code></pre>
<p>and am getting the following error:</p>
<pre><code class="lang-auto">TypeError: forward() missing 1 required positional argument: 'target'
</code></pre>
<p>I’ve done nothing to alter the forward method at all, so I’m quite confused. I’d appreciate any help. Thank you!</p>
<p>The full code for my class is below:</p>
<pre><code class="lang-auto">class Network(nn.Sequential):
    def __init__(self, layers):
        super().__init__(self.init_modules(layers))
        self.criterion = nn.NLLLoss()
        self.optimizer = optim.Adam(self.parameters(), lr=0.003)


    def train(self, trainloader, epochs):
        for e in range(epochs):
            for x, y in trainloader:
                x = x.view(x.shape[0], -1)
                self.optimizer.zero_grad()
                loss = self.criterion(self(x), y)
                loss.backward()
                self.optimizer.step()

    def init_modules(self, layers):
        # Logic unimportant to the question (I think)
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>The problem is that <code>nn.Sequential</code>, gets all the modules in your network and pass your input in <code>.forward</code> to all of those modules. Currently, <code>self.criterion</code> is a module and it is the first module. if you print this code, you get list of modules:</p>
<pre><code class="lang-auto">net = Network()
list(net.modules())

# [Network(
#   (criterion): NLLLoss()
# ), NLLLoss()]
</code></pre>
<p>which as you can see, <code>nn.Sequential</code> pass <code>x</code> to <code>NLLLoss</code> and obviously this module needs a <code>target</code> too! And after this, next module will be <code>optimizer</code> which does not accept any input at all.</p>
<p>The problem is you are defining a full model(loss, optimizer, etc) by extending <code>nn.Sequential</code> which is not correct. If you want to extend <code>nn.Sequential</code> you need to follow its own patterns.</p>
<p>Here is an instance:</p>
<pre><code class="lang-auto">class Network(nn.Sequential):
    def __init__(self, layers=(nn.Conv2d(1, 3, 3), nn.Conv2d(3, 10, 3)), act='relu'):
        super(Network, self).__init__()
        self.init_modules(layers, act)
    def init_modules(self, layers, act):
        for idx, module in enumerate(layers):
            self.add_module(str(idx), module)
            if act == 'relu':
                self.add_module(str(idx)+'act', nn.ReLU())
        
</code></pre>
<p>Module list in this case:</p>
<pre><code class="lang-auto">[Network(
   (0): Conv2d(1, 3, kernel_size=(3, 3), stride=(1, 1))
   (0act): ReLU()
   (1): Conv2d(3, 10, kernel_size=(3, 3), stride=(1, 1))
   (1act): ReLU()
 ),
 Conv2d(1, 3, kernel_size=(3, 3), stride=(1, 1)),
 ReLU(),
 Conv2d(3, 10, kernel_size=(3, 3), stride=(1, 1)),
 ReLU()]
</code></pre>
<p>PS. note the line with <code>super</code>, you have to use same structure. <code>super(MyModule, self).__init__()</code></p>
<p>Bests</p>

username: Wesley_Neill
cooked: <p>I have to admit that I’m still confused. I’m sorry, I’m just learning OOP in python. I want my extended class to have two extra attributes that are not modules for feedforward.</p>
<p>Those two attributes are the optimizer and the loss criterion. I don’t want them to be part of the layer/module pipeline. I just want them as attributes for use in my training method.</p>
<p>An additional problem is that I don’t want to have to initialize the the object with pre-existing modules. Rather, I want to declare the object with the <code>layers</code> tuple, and have the modules dynamically generated the modules based off that tuple. In your example, you have initialized the object with two <code>Conv2d</code> layers, which is the opposite of what I am trying to achieve.</p>
<p>Where can I declare attributes so that they DO NOT end up as modules? I thought all unique child attributes were supposed to be declared in the <code>__init__</code> method. After declaring the unique child attributes, I thought you called <code>super().__init__()</code> in order to have the parent class finish filling out the object.</p>

username: Nikronic
cooked: <p>Can you use a psuedocode or a python code to show wha you mean by this, I cannot figure it out by tuple of layers, what are the values in tuple?</p>
<aside class="quote no-group" data-username="Wesley_Neill" data-post="3" data-topic="85848">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p>I don’t want to have to initialize the the object with pre-existing modules. Rather, I want to declare the object with the <code>layers</code> tuple, and have the modules dynamically generated the modules based off that tuple.</p>
</blockquote>
</aside>

username: Wesley_Neill
cooked: <p>I was able to get my code to work removing <code>self.optimizer = optim.Adam()</code> and <code>self.criterion = nn.NLLLoss()</code> in the constructor, as you suggested. I had to make them normal variables in the training method rather than class attributes. That’s a little bit of a bummer, but at least I have working code <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>To answer your question:</p>
<p>The ‘layers’ tuple is one where each value in the tuple is the number of nodes for that corresponding layer:</p>
<p><code>layers = (n_nodes_layer1, n_nodes_layer2, ...)</code></p>
<p>So now, the below code works fine. I had to move the optimizer and the criterion into the training method as regular variables rather than as class attributes. It is less flexible this way, but works now:</p>
<pre><code class="lang-auto">class LSequential(nn.Sequential):
    def __init__(self, layers):
        super().__init__(self.init_modules(layers))

    def init_modules(self, layers):

        n_layers = len(layers)
        modules = OrderedDict()

        # Layer definitions for input and inner layers:
        for i in range(n_layers - 2):
            modules[f'fc{i}'] = nn.Linear(layers[i], layers[i + 1])
            modules[f'relu{i}'] = nn.ReLU()

        # Definition for output layer:
        modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
        modules['smax_out'] = nn.LogSoftmax(dim=1)

        return modules
</code></pre>

username: Nikronic
cooked: <aside class="quote no-group" data-username="Wesley_Neill" data-post="5" data-topic="85848">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p><code>nn.Linear(layers[i], layers[i + 1])</code></p>
</blockquote>
</aside>
<p>Now that makes sense. I was thinking how can one create a network combined of Conv1d and Conv2d and few other layers using just tuple of numbers of in out channels.</p>
<p>Let me disagree with you on this topic. Actually, one of the most fascinating things about PyTorch is its modularity and in this case, you are building a customized model, a network which will be represented as a computational graph. This model can be used for different tasks with different purposes. So, embedding loss and optimizer within train method makes it less modular. For instance if I want to use your implementation for my own data, I had to change source code directly rather than passing the final model and its outputs to a customized loss or optimizer.<br>
If you go through more in PyTorch’s source code, you will enjoy its modularity!</p>
<aside class="quote no-group" data-username="Wesley_Neill" data-post="5" data-topic="85848">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p>It is less flexible this way, but works now</p>
</blockquote>
</aside>
<p>If you are interested in creating high level classes and wrapping all those steps into single class, I think reading source code of PyTorch Lightening and Ignite would be a good idea.</p>

username: Wesley_Neill
cooked: <p>Agreed! As i’m learning, I’m being astounded by the flexibility of PyTorch. You can customize everything (I don’t know much a bout neural nets yet, so I’m sure I’ll continued to be blown away!)</p>
<p>I meant flexibility in programming style. I’m coming from a Java background, where the parent cannot see child specific attributes unless they are past to <code>super()</code>. That means I have to tip-toe around with child/parent relationships because unless I REALLY know what’s going on in the parent, I can’t trust it to not mess with the child in unexpected ways.</p>
<p>But, Python is not Java, so I’m okay with learning a new style. And this inheritance weirdness spurred me to look at the source code for <code>nn.Sequential</code>, and for a change I actually was able to understand what was going on. Normally, source code only confuses me more.</p>
<p>All in all, PyTorch has been an awesome experience.</p>