How to define the activation function ReLU(x) * ReLU(1-x)?

Kejie_Tang: <p>I want to define the activation function ReLU(x) * ReLU(1-x). But I only know this type:</p>
<pre><code class="lang-auto">class Act_op(nn.Module):
    def __init__(self):
        super(Act_op, self).__init__()
    def forward(self, x):
        return x ** 50
class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.block = nn.Sequential()
        for i in range(len(R_variable['full_net'])-2):
            self.block.add_module('linear'+str(i), nn.Linear(R_variable['full_net'][i],R_variable['full_net'][i+1]))
            self.block.add_module('**50'+str(i), Act_op())
        i = len(R_variable['full_net'])-2
        self.block.add_module('linear'+str(i), nn.Linear(R_variable['full_net'][i],R_variable['full_net'][i+1]))
    def forward(self, x):
        out = self.block(x)
        return out
</code></pre>
<p>Thanks a lot!!</p>
<p>My code’s purpose is that when I input [1,100,100,1], it creates a DNN of which the structure is (linear(1,100), relu(), linear(100,100), relu(), linear(100,1)). (The example replace relu() with x**50). The net is adaptive. Therefore I use  <code>nn.Sequential.add_module()</code> . Now I want to replace relu() with relu(x)*relu(1-x). I don’t know how to do it.</p>

KFrank: <p>Hello Kejie!</p>
<aside class="quote no-group" data-username="Kejie_Tang" data-post="1" data-topic="69049" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kejie_tang/48/12395_2.png" class="avatar"> Kejie_Tang:</div>
<blockquote>
<p>I want to define the activation function ReLU(x) * ReLU(1-x).</p>
</blockquote>
</aside>
<p>The short answer is that you just do.</p>
<p><code>torch.nn.functional.relu()</code> (and its class version, <code>torch.nn.ReLU</code>)<br>
is differentiable (in the pytorch sense), so its product is as well, and both<br>
<code>relu()</code> and its product work just fine with autograd and <code>backward()</code>.</p>
<p>I don’t understand the point of the code you posted, nor its relevance<br>
to the question in the title of your post, but, quite simply:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
one = torch.autograd.Variable (torch.FloatTensor ([1.0]))
x = one / 2
torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
x = -one
torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
x = 10 * one
torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
</code></pre>
<p>(The nonsense with <code>autograd.Variable</code> is because I’m using pytorch<br>
version 0.3.0.)</p>
<p>Here are the results:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; one = torch.autograd.Variable (torch.FloatTensor ([1.0]))
&gt;&gt;&gt; x = one / 2
&gt;&gt;&gt; torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
Variable containing:
 0.2500
[torch.FloatTensor of size 1]

&gt;&gt;&gt; x = -one
&gt;&gt;&gt; torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
Variable containing:
 0
[torch.FloatTensor of size 1]

&gt;&gt;&gt; x = 10 * one
&gt;&gt;&gt; torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
Variable containing:
 0
[torch.FloatTensor of size 1]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Kejie_Tang: <p>Thank a lot! I think my code is a bit confusing. My code’s purpose is that when I input [1,100,100,1], it creates a DNN of which the structure is (linear(1,100), relu(), linear(100,100), relu(), linear(100,1)). (The example replace relu() with x**50). The net is adaptive. Therefore I use <code>nn.Sequential.add_module()</code>.  Now I want to replace relu() with relu(x)*relu(1-x). I don’t know how to do it. I’m sorry I didn’t clarify the problem.</p>