Constrained PReLU

FrankiePoon: <p>I would like to use a 1-Lipschitz continuous function as an activation function in my network, such as ReLU and LeakyReLU (with negative slop between 0 and 1).</p>
<p>I also would like to use an activation function similar to PReLU with a learnable parameter while having the 1-Lipschitz continuous property. Are there any ways that I can add a constraint to the torch.nn.PReLU such that the learnable parameter is bound between 0 and 1?</p>

KFrank: <p>Hi Frankie!</p>
<aside class="quote no-group" data-username="FrankiePoon" data-post="1" data-topic="173152" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/frankiepoon/48/56313_2.png" class="avatar"> FrankiePoon:</div>
<blockquote>
<p>Are there any ways that I can add a constraint to the torch.nn.PReLU such that the learnable parameter is bound between 0 and 1?</p>
</blockquote>
</aside>
<p>Probably the best way is to train an unbounded parameter, but transform<br>
it into a bounded prelu-weight with <code>sigmoid()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1'
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt; wt = torch.randn (1, requires_grad = True)   # learnable unbounded weight
&gt;&gt;&gt; wt_bnd = torch.sigmoid (wt)                  # derived bounded weight
&gt;&gt;&gt; loss = torch.nn.functional.prelu (torch.randn (5), wt_bnd).sum()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; wt.grad
tensor([-0.1999])
</code></pre>
<p><code>wt_bnd</code>, the <code>weight</code> parameter passed to the functional version of <code>prelu()</code>,<br>
is bounded between 0 and 1, while the actual trainable parameter, <code>wt</code>, is<br>
unbounded, so you can train it without needing to add constraints somehow<br>
to the optimizer.</p>
<p>Best.</p>
<p>K. Frank</p>

FrankiePoon: <p>Thanks for your suggestions. To me, it is a normalization of the learnable parameter, but can we say now our activation function is 1-Lipschitz continuous?</p>

KFrank: <p>Hi Frankie!</p>
<aside class="quote no-group" data-username="FrankiePoon" data-post="3" data-topic="173152" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/frankiepoon/48/56313_2.png" class="avatar"> FrankiePoon:</div>
<blockquote>
<p>can we say now our activation function is 1-Lipschitz continuous?</p>
</blockquote>
</aside>
<p>I’m not sure what you are asking.</p>
<p>If you’re asking whether our “constrained prelu” is Lipschitz continuous<br>
when viewed as a function of just its <code>input</code> (with its <code>weight</code> parameter<br>
held fixed), the answer is yes.  By inspection <code>prelu (input)</code> is Lipschitz<br>
continuous.</p>
<p>If your asking whether <code>prelu (input, weight)</code> is Lipschitz continuous<br>
when understood as a function of two variables, <em>even though the <code>weight</code><br>
we pass in is bounded,</em> the answer is no.  For <code>input &lt; 0</code>,<br>
<code>prelu (input, weight) = weight * input</code>, so the partial derivative,<br>
<code>d prelu (input, weight) / d weight  =  input</code>, is unbounded<br>
because <code>input</code> can run off to <code>inf</code> and <code>-inf</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

FrankiePoon: <p>Hi Frank,</p>
<p>Thanks for your answers. I was trying to ask whether our “constrained prelu” is Lipschitz continuous. It is all clear to me now.</p>
<p>Thanks for all your supports.</p>
<p>Best,</p>
<p>Frankie</p>