Linear activation function

H_MP: <p>Hi, I am training a custom CNN, I need to use a linear activation function. but I didn’t find anything in pytorch. I khow this activation just pass the input to the output of it, so should I use <code>nn.Linear(nin, nin)</code> or <code>nn.Identity()</code> or do nothing?<br>
while I am training my network, the training and validation is nearly constant and I think this is cause of bad usage of my activation functions<br>
Here is my model:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f30d2eeb60835d1491ecd390e77d6c05f2da4abc.png" data-download-href="https://discuss.pytorch.org/uploads/default/f30d2eeb60835d1491ecd390e77d6c05f2da4abc" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f30d2eeb60835d1491ecd390e77d6c05f2da4abc_2_690x227.png" alt="1" data-base62-sha1="yG8gLhn7JuYWJUB23zz9atK1ZDu" width="690" height="227" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f30d2eeb60835d1491ecd390e77d6c05f2da4abc_2_690x227.png, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f30d2eeb60835d1491ecd390e77d6c05f2da4abc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f30d2eeb60835d1491ecd390e77d6c05f2da4abc.png 2x" data-dominant-color="E7A39C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1014×335 70.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18ed0b74695f28426315f158efe72afbb7959667.png" data-download-href="https://discuss.pytorch.org/uploads/default/18ed0b74695f28426315f158efe72afbb7959667" title="2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18ed0b74695f28426315f158efe72afbb7959667.png" alt="2" data-base62-sha1="3yvkavCHNgKq7UFuUE1L3YJmOA7" width="690" height="255" data-dominant-color="E5E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">860×319 37.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41e4fc4abe996bf8db9b5b3328374af65e3e5b21.png" data-download-href="https://discuss.pytorch.org/uploads/default/41e4fc4abe996bf8db9b5b3328374af65e3e5b21" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41e4fc4abe996bf8db9b5b3328374af65e3e5b21_2_690x431.png" alt="3" data-base62-sha1="9oVDm6H2JDNWuux2r5vlD0s0esx" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41e4fc4abe996bf8db9b5b3328374af65e3e5b21_2_690x431.png, https://discuss.pytorch.org/uploads/default/original/3X/4/1/41e4fc4abe996bf8db9b5b3328374af65e3e5b21.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/41e4fc4abe996bf8db9b5b3328374af65e3e5b21.png 2x" data-dominant-color="F0F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">905×566 192 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and here is my code:</p>
<pre><code class="lang-auto">class Block(nn.Module):
    def __init__(self, in_channels, out_channels, exp=1, stride=1, type=''):
        super().__init__()
        self.t = type
        self.stride = stride
        self.inc, self.outc = in_channels, out_channels
        self.exp = exp

        
        self.blockc = nn.Sequential(
            nn.Conv2d(self.inc, self.inc* self.exp, kernel_size=1),
            nn.ReLU6(inplace=True), 
            nn.Conv2d(self.inc * self.exp, self.inc * self.exp, kernel_size=3, groups= self.inc * self.exp, stride= self.stride, padding=1),
            nn.ReLU6(inplace=True),
            nn.Conv2d(self.inc * self.exp, self.outc, kernel_size=1),
            nn.Identity())#,
        
    def forward(self, x):
        out = self.blockc(x)
        
        if self.t == 'A':
            out = torch.add(out,x)

        return out



class Model(nn.Module):
    def __init__(self):
        
        super().__init__()
        self.conv2d1 = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3,padding=1, stride=2)
        self.r = nn.ReLU6()
        self.stage1 = Block(8, 8, exp=1, stride=2, type='C')
        self.stage2 = nn.Sequential(
            Block(8, 16, exp=2, stride=2, type='C'),
            Block(16, 16, exp=2, type='A'))
        self.stage3 = nn.Sequential(
            Block(16, 24, exp=2, stride=2, type='C'),
            Block(24, 24, exp=2, type='A'))
        self.post_block2 = Block(24, 32, exp=2, type='B')

        self.fc = nn.Linear(128, 10)
    def forward(self, x):
        out = self.conv2d1(x)
        out = self.r(out)
        out = self.stage1(out)
        out = self.stage2(out)
        out = self.stage3(out)
        out = self.post_block2(out)

        out = out.view(-1, 128)

        out = self.fc(out)
        return out
</code></pre>

JuanFMontesinos: <p>Can you define that mathematically?<br>
The concept itself sounds like you wanna do y=mx+n where m=1 and n=0, so basically the identity operator.</p>

AlphaBetaGamma96: <p>Following on from what <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> has already stated, <a class="mention" href="/u/h_mp">@H_MP</a> please share some code so we can better understand your problem. Usually, with a ‘linear’ activation function, you can just “do nothing” and return the input and that’s fine.</p>
<p>But do share some code (and wrap it in 3 backticks ``` to get the correct indentation) so your problem can be solved.</p>

H_MP: <p>Here <code>exp</code> means that the output of the first Conv1x1 in <code>Block</code> class is =&gt; <code>out_channel = in_channel * exp</code></p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="H_MP" data-post="1" data-topic="158275">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/41988e/48.png" class="avatar"> H_MP:</div>
<blockquote>
<pre><code class="lang-auto">        super().__init__()
</code></pre>
</blockquote>
</aside>
<p>Make sure to properly define your parent classes as that can cause problems as your class isn’t technically a <code>nn.Module</code>.</p>
<p>Your <code>nn.Identity()</code> question is about the <code>self.blockc</code> defintion?</p>

H_MP: <p>Thank for your reply, Yes, I want to use Linear activation function, so I don’t know how to use it</p>

AlphaBetaGamma96: <p>I think you can just remove the <code>nn.Identity</code> function as there’s no need for it.</p>
<p>Also make sure to properly define your parent class, i.e. change <code>super().__init__()</code> to <code>super(Block, self).__init__()</code> for <code>Block</code> and the same for <code>Model</code>.</p>

H_MP: <p>actually, removing the <code>nn.Identity</code> solved my problem. Thanks a lot</p>