Binary/Piecewise activation function

username: ily83
cooked: <p>Hello, how can I create a custom activation function like the binary step for example:<br>
binary = lambda x: np.where(x&gt;=0, 1, 0) ?</p>
<p>I tried “activation = lambda x: torch.where(x &lt; 0.5, 1., 0.)” and i run into an error:</p>
<blockquote>
<p>runtimeError                              Traceback (most recent call last)<br>
/tmp/ipykernel_615/1226976390.py in <br>
69             x_colloc_tens = torch.tensor(x_colloc,requires_grad=True).float().to(device)<br>
70             t_colloc_tens = torch.tensor(t_colloc,requires_grad=True).float().to(device)<br>
—&gt; 71             f_out = f(x_colloc_tens,t_colloc_tens)<br>
72             zeros_tens = torch.tensor(np.zeros((taille_dataset,1)),requires_grad=True).float().to(device)</p>
<p>/tmp/ipykernel_615/828930035.py in f(x, t)<br>
8    u = modele(entry_data)<br>
----&gt; 9     u_x = torch.autograd.grad(u, x, torch.ones_like(u),retain_graph=True,create_graph=True)[0]<br>
10     u_t = torch.autograd.grad(u, t, torch.ones_like(u),retain_graph=True,create_graph=True)[0]</p>
<p>~/.conda/envs/default/lib/python3.9/site-packages/torch/autograd/<strong>init</strong>.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)<br>
232         retain_graph = create_graph<br>
233<br>
 → 234     return Variable.<em>execution_engine.run_backward(<br>
235         outputs, grad_outputs</em>, retain_graph, create_graph,<br>
236         inputs, allow_unused, accumulate_grad=False)</p>
<p>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
</blockquote>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/ily83">@ily83</a><br>
We can use it like this</p>
<pre><code class="lang-auto">class custom_activation(nn.Module):
    def __init__(self):
        super(custom_activation, self).__init__()
    
    def forward(self, x):
        x[x&gt;=0] = 1
        x[x&lt;0] =0
        return x

class random_model(nn.Module):
    def __init__(self, num_layers):
        super(random_model, self).__init__()
        self.layer1 = nn.Linear(100, 20)
        self.layer2 = nn.Linear(20, 10)
        self.layer3 = nn.Linear(10, 1)
        self.custom = custom_activation()
        self.sigm = nn.Sigmoid()
    
    def forward(self, x):
        x = self.custom(self.layer1(x))
        x = self.layer2(x)
        x = self.custom(x)
        x = self.layer3(x)
        x = self.sigm(x)
        return x
</code></pre>

username: KFrank
cooked: <p>Hi Bou!</p>
<aside class="quote no-group" data-username="ily83" data-post="1" data-topic="143516" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ea5d25/48.png" class="avatar"> ily83:</div>
<blockquote>
<p>I tried “activation = lambda x: torch.where(x &lt; 0.5, 1., 0.)”</p>
</blockquote>
</aside>
<p>Please note that such an activation is almost certainly not what you want<br>
because it is not <em>usefully</em> differentiable.  That is, its derivative is zero<br>
everywhere (except at <code>x = 0.5</code> where, technically, its derivative is not<br>
defined).</p>
<p>So even if you write a version that supports pytorch’s autograd automatic<br>
differentiation, such as by using the approach that Anant suggested, any<br>
gradients you try to backpropagate through your custom activation function<br>
will become zero.</p>
<p>(If you want to backpropagate through a step-like function, you would<br>
typically use a “soft” step function such as <code>sigmoid()</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>