How to add a layer to an existing Neural Network?

Fractale: <p>actually I use:<br>
<code>torch.nn.Sequential(model, torch.nn.Softmax())</code><br>
but It create a new sequence with my model has a first element and the sofmax after. It’s not adding the sofmax to the model sequence.<br>
I know these 2 networks will be equivalenet but I feel it’s not really the correct way to do that.</p>

ptrblck: <p>It should generally work.<br>
Here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(10, 2)
        
    def forward(self, x):
        x = self.fc(x)
        return x


model = MyModel()
x = torch.randn(1, 10)
print(model(x))
&gt; tensor([[-0.2403,  0.8158]], grad_fn=&lt;ThAddmmBackward&gt;)

model = nn.Sequential(
    model,
    nn.Softmax(1)
)
print(model(x))
&gt; tensor([[0.2581, 0.7419]], grad_fn=&lt;SoftmaxBackward&gt;)
</code></pre>
<p>As you can see, the output was normalized using softmax in the second call.<br>
Also the <code>grad_fn</code> points to softmax.<br>
Could you print your model after adding the softmax layer to it?</p>

Fractale: <p>Sorry I was probably not clear.<br>
This is how I create my model.<br>
But when I print my model, it’s a model inside a model, inside a model, inside a model, not a list of layers.<br>
Is there a better way to do that?</p>
<pre><code class="lang-auto">def getMultiLayerPerceptron(InputNetworkSize, NbClass, nbHidenLayer,
                                   hidden_dimension_size, activationFunction):
    model = torch.nn.Sequential(
        torch.nn.Linear(InputNetworkSize, hidden_dimension_size),
        activationFunction())
    for i in range(nbHidenLayer - 1):
        model = torch.nn.Sequential(
            model,
            torch.nn.Linear(hidden_dimension_size, hidden_dimension_size),
            activationFunction())

    model = torch.nn.Sequential(
        model, torch.nn.Linear(hidden_dimension_size, NbClass))
    return model
</code></pre>
<p>A use <code>torch.nn.Sequential</code> because I don’t understand what should I put in the <code>__init__</code> and what should I put in the <code>forward</code> function when using a class for a multi-layer fully connected neural network</p>

ptrblck: <p>Well, you could also define these layers inside the <code>__init__</code> of another module.<br>
Here is an example using <code>nn.ModuleList</code>:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, in_features, nb_classes, nb_hidden_layer, 
        hidden_size, act=nn.ReLU):
        super(MyModel, self).__init__()
        self.act = act()
        
        self.fc1 = nn.Linear(in_features, hidden_size)
        self.fcs = nn.ModuleList([nn.Linear(hidden_size, hidden_size)])
        self.out = nn.Linear(hidden_size, nb_classes)
        
    def forward(self, x):
        x = self.act(self.fc1(x))
        for l in self.fcs:
            x = F.relu(l(x))
        x = self.out(x)
        return x
            
model = MyModel(2, 3, 4, 5, nn.ReLU)
</code></pre>
<p>You could also use <code>nn.ModuleDict</code> to set the layer names.</p>

pulpaul: <p>And how do you add a Fully Connected layer to a Pretrained ResNet50 Network?</p>

ptrblck: <p>I assume you would like to add the new linear layer at the end of the model?<br>
If so, <code>resnet50</code> uses the <code>.fc</code> attribute to store the last linear layer:</p>
<pre><code class="lang-python">model = models.resnet50()
print(model.fc)
&gt; Linear(in_features=2048, out_features=1000, bias=True)
</code></pre>
<p>You could store this layer and add a new <code>nn.Sequential</code> container as the <code>.fc</code> attribute via:</p>
<pre><code class="lang-python">lin = model.fc
new_lin = nn.Sequential(
    nn.Linear(lin.in_features, lin.in_features),
    nn.ReLU(),
    lin
)

model.fc = new_lin
</code></pre>

pulpaul: <p>And Do I need to modify the forward function on the model class? Thanks</p>

ptrblck: <p>If you replace an already registered module (e.g. <code>model.fc</code>), you would have to make sure that the setup (expected input and output shapes) are valid. Other than that, you wouldn’t need to change the <code>forward</code> method and this module will still be called as in the original <code>forward</code>.</p>
<p>However, if you need to add changes, which aren’t a simple replacement of layers, I would recommend to manipulate the <code>forward</code> method.</p>

gyt.971027: <p>What should I do to add quant and dequant layer in a pre-trained model?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html">BERT quantization tutorial</a> seems to load a pr-trained model and apply dynamic quantization to it, so it could be helpful.</p>

HoaAnDuong: <p>If this discuss page have an upvote system, i will give a upvote for u</p>