Fusing of Layers

Ganga: <ul>
<li>
<p>is there any way possible to do fusing on FC layers</p>
</li>
<li>
<p>Can we do do fusing of Bnorm , when Linear/Conv followed by BnormI.</p>
</li>
</ul>

JuanFMontesinos: <p>Fusing? What do you mean?<br>
Using 2 FC layers with no non-linearities in between is equivalent to using a single FC as they are linear.</p>

Ganga: <p>here is the code ,i want to fuse the FC3 layer(last layer which is linear) with bnorm(nn.BatchNorm1d) is that possible?</p>
<pre><code class="lang-auto">
    def __init__(self, num_classes=1000):
        super(AlexNet, self).__init__()
        self.num_classes = num_classes
        self.features = nn.Sequential(
            nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=0),
            nn.BatchNorm2d(96, eps=1e-4, momentum=0.1, affine=True),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            BinConv2d(96, 256, kernel_size=5, stride=1, padding=2, groups=1),
            nn.MaxPool2d(kernel_size=3, stride=2),
            BinConv2d(256, 384, kernel_size=3, stride=1, padding=1),
            BinConv2d(384, 384, kernel_size=3, stride=1, padding=1, groups=1),
            BinConv2d(384, 256, kernel_size=3, stride=1, padding=1, groups=1),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            BinConv2d(256 * 6 * 6, 4096, Linear=True),
            BinConv2d(4096, 4096, dropout=0.5, Linear=True),
            nn.BatchNorm1d(4096, eps=1e-3, momentum=0.1, affine=True),
            nn.Dropout(),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 256 * 6 * 6)
        x = self.classifier(x)
        return x```</code></pre>

JuanFMontesinos: <p>Well, still don’t understand what do you mean by fuse but yes. You can use batchnorm after a linear layer if the output is a 2D tensor.</p>
<p>Think that Pytorch’s implementation of Linear allows to use N-Dimensional tensors.</p>

Ganga: <p>I want to fuse bnorm weight with linear weight ,where linear follows the bnorm(bnorm comes first ,then linear next). is it possible?<br>
can you provide the sequences possible for fusing?</p>

JuanFMontesinos: <p>Something like the following?</p>
<pre><code class="lang-auto">import torch
from torch import nn


class BNLinear:
    def __init__(self, in_features: int, out_features: int, bias: bool = True):
        self.linear = nn.Linear(in_features, out_features, bias)
        self.batch_norm = nn.BatchNorm1d(in_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

    def forward(self, x):
        x = self.batch_norm(x)
        return self.linear(x)
</code></pre>

Ganga: <p>I have a Relu in between 2FC as non-linearities ,in that case can we fuse the linear wegihts with bnorm weights</p>

JuanFMontesinos: <p>Well not in principle. The weights itself are dedicated to learn different stuff. There are not even the same amount of parameters.</p>

Ganga: <p>Are you talking about the 2FC layers?Can you be please more specific,I have struck in this for long time. your help in this are more appreciated</p>

Ganga: <p>1.Basically i am replicating the software in hardware(verilog coding),so to code Bnorm in hardware its very complicated</p>
<p>2.So i have fused(learnable parametrs) the Bnorm of my first layer with Conv (Conv-&gt;Bnorm)<br>
3. For FC I have Bnorm  in 3rd FC layer, I want to avoid these in my model, so it will be useful to code in hardware<br>
4. So thought of fusing Bnorm with  near by layers<br>
5.My FC2 and FC3 layers looks like (Linear-&gt;Relu-&gt;Bnorm-&gt;dropout-&gt;Linear)<br>
6.I tried to fuse (Linear-&gt;Relu-&gt;<em><strong>Bnorm</strong></em>-&gt;dropout-&gt;<strong>Linear</strong>) it didn’t worked out due to its shape<br>
7. So I tried to fuse (<strong>Linear</strong>-&gt;Relu-&gt;<em><strong>Bnorm</strong></em>-&gt;dropout-&gt;Linear), I fused it but  prediction are not correct by taking fused weights and bias<br>
8.<strong>So I want to try to fuse(learnable parameter) (<strong>Linear-&gt;Relu</strong>) and then fused result again to fuse with Bnorm</strong><br>
9.Is that possible to do ?<br>
Your help will be appreciated</p>

RaLo4: <p>What you mean by <strong>fuse</strong> is very confusing and unclear.<br>
For example:</p>
<aside class="quote no-group" data-username="Ganga" data-post="10" data-topic="93088">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ganga/48/28549_2.png" class="avatar"> Ganga:</div>
<blockquote>
<ol start="8">
<li><strong>So I want to try to fuse(learnable parameter) ( <strong>Linear-&gt;Relu</strong> )</strong></li>
</ol>
</blockquote>
</aside>
<p>Is very confusing, since the normal ReLU is a classical activation function defined as <code>ReLU(x)=max(0,x)</code>  and has no learnable parameters (unlike activation functions like PReLU).</p>
<p>I think everything that can be said was already said by <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a><br>
If this is not the answer you are looking for</p>
<aside class="quote no-group" data-username="JuanFMontesinos" data-post="6" data-topic="93088" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>Something like the following?</p>
<pre><code class="lang-auto">import torch
from torch import nn


class BNLinear:
    def __init__(self, in_features: int, out_features: int, bias: bool = True):
        self.linear = nn.Linear(in_features, out_features, bias)
        self.batch_norm = nn.BatchNorm1d(in_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

    def forward(self, x):
        x = self.batch_norm(x)
        return self.linear(x)
</code></pre>
</blockquote>
</aside>
<p>then this is probably your answer:</p>
<aside class="quote no-group" data-username="JuanFMontesinos" data-post="8" data-topic="93088" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>Well not in principle. The weights itself are dedicated to learn different stuff. There are not even the same amount of parameters.</p>
</blockquote>
</aside>

Ganga: <ul>
<li>
<p>My main Intention is  to fuse bnorm, (linear-&gt;relu-&gt;bnorm), but to fuse bnorm with linear,in between i have relu,so i thought to fuse linear and relu and then with bnorm</p>
</li>
<li>
<p>ok , I got it Relu doesn’t have learnable parameters,but can I fuse Linear with Relu as I saw its possible sequence to fuse</p>
</li>
<li>
<p>If it so,Can you provide the code to fuse</p>
</li>
</ul>

RaLo4: <p>I still don’t think I understand you.<br>
But if you just want to have Linear and ReLU in sequence and callable in one, than this works:</p>
<pre><code class="lang-auto">import torch.nn as nn

Linear_ReLU_fusion = nn.Sequential(nn.Linear(in_features, out_features),
                                   nn.ReLU()
                                   )
</code></pre>
<p>you could then just call it in one go</p>
<pre><code class="lang-auto">out = Linear_ReLU_fusion(x)
</code></pre>
<p>e.g.</p>
<pre><code class="lang-auto">import torch.nn as nn

Linear_ReLU_fusion = nn.Sequential(nn.Linear(3072, 10),
                                   nn.ReLU()
                                   )
x = torch.rand(1, 3, 32, 32)
x = x.view(-1, 3072)
out = Linear_ReLU_fusion(x)
</code></pre>
<p>works fine!</p>

Ganga: <p>Can I do this to the module wrapped in an sequential?</p>

ptrblck: <p>I don’t really understand the last question.</p>
<p>However, as was already said, you can build “blocks” e.g. via <code>nn.Sequential</code> and use it to call the linear layer with a relu activation function.</p>
<p>That being said, it’s not a “fusion” but just a way to write a model.</p>
<p>I’m still unsure, if you want to fuse mathematical operations, e.g. two linear layers:</p>
<pre><code class="lang-python"># sequential approach
y1 = x * w1 + b1
y2 = y1 * w2 + b2

# "fused"
y2 = (x * w1 + b1) * w2 + b2
y2 = x * w1 * w2 + b1 * w2 + b2
y2 = x * w3 + b3
</code></pre>
<p>where <code>w3 = w1 * w2</code> and <code>b3 = b1 * w2 + b2</code>, which wouldn’t be possible, if you have a <code>relu</code> between these layers, or if you are talking about kernel fusion, where you are writing e.g. CUDA kernels to perform multiple operations in a single kernel.</p>

Ganga: <ul>
<li>
<p>that’s what I exactly needed,fusing weights and biases</p>
</li>
<li>
<p>in FC2 layer I have(linear-&gt;relu),In FC3 I have(Bnorm-&gt;linear)</p>
</li>
<li>
<p>I tried fusing <em>FC2</em> <strong>linear</strong> with FC3 <strong>Bnorm</strong> but fused weights and bias are not correct as my prediction was wrong</p>
</li>
<li>
<p>From your point if we have <strong>relu</strong> in between we cant fuse, OK i agreed</p>
</li>
<li>
<p>So i am trying to fuse <strong>FC3 Bnorm</strong> with <strong>FC3 Linear</strong>(Bnorm-&gt;linear) will i able to do it?</p>
</li>
<li>
<p>I am using this formula <img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e337d2d15df63aef41c265cb848ded57c800b956.jpeg" alt="formula" data-base62-sha1="wq3ZcqDAPIkKHp2kxeLaTLzImOO" width="564" height="202"></p>
</li>
</ul>

ptrblck: <p>I don’t see how this would be possible, but let’s have a loop at the operations:</p>
<pre><code class="lang-python"># sequential
y1 = x * w1 + b1 # linear
y2 = (y1 - running_mean) / sqrt(running_var + eps) * gamma + beta # batchnorm

# replace y1
y2 = (x * w1 + b1 - running_mean) / sqrt(running_var + eps) * gamma + beta
</code></pre>
<p>If you multiply gamma into the left brackets, you would mix the <code>running_mean</code> with a trainable parameter, so how should this “fusion” look like?</p>

Ganga: <ul>
<li>
<p>I want to know this example what u have gave is 1st is linear and then Bnorm?</p>
</li>
<li>
<p>Will be any change if BNorm comes 1st and then Linear?</p>
</li>
</ul>

Yadu_Krishnan_S: <p>In your explaination, you have shown with Linear followed by batchnorm.<br>
Can we fuse even if batchnorm comes first followed by Linear</p>

Ganga: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,I have fused linear and bnorm using above formula,I am able to do it,prediction is correct by using fused weights and bias. I am getting all values <strong>negative</strong> (<strong>output of fused</strong> <strong>layer</strong>),that is the linear layer(<strong>last layer</strong>),can it be possible to get all values negative?</p>