Efficiently applying per neuron activation functions

ParticularlyPythonic: <p>I want to use a custom activation function that has a random component that gets applied to every neuron individually.</p>
<p>If I use the standard method and call the activation function on a layer, it applies the same value to every neuron in that layer.</p>
<p>I am looking for the most efficient way to have the activation function affect every neuron individually and would appreciate any advise on the topic.</p>
<p>The only method I currently know about involves splitting the input tensor and applying the function each part and then concatenating the result, but I am hoping there is a more efficient way</p>

albanD: <p>Hi,</p>
<p>That will depend a lot on what your function is. Can you be more precise what it is? Or provide a code sample that shows the split/concat version?</p>

ParticularlyPythonic: <p>The function is going to be a modified mish with a random coefficient in the exponential.</p>
<p>The split concat version I was talking about was referencing this forum post: <a href="https://discuss.pytorch.org/t/how-to-apply-different-activation-fuctions-to-different-neurons/13475" class="inline-onebox">How to apply different activation fuctions to different neurons</a></p>

albanD: <p>If you want a modified mish, then I would implement the function using just regular element-wise functions. And then you create a Tensor with the coefficient you want and can add/multiply it at any place your need in the computation.</p>

ParticularlyPythonic: <p>I am sorry, I should have been more clear about the activation function</p>
<pre><code class="lang-auto">x *( torch.tanh(F.softplus(x, beta = random.uniform(self.beta_lower, self.beta_higher))))
</code></pre>
<p>This is the function I want to use. And I want to know the best way to get a different beta for each neuron.</p>
<p>Thanks for your patient response!</p>

albanD: <p>Hi,</p>
<p>I’m afraid the softplus function only accepts a single beta value. But you can just re-implement it</p>
<pre><code class="lang-python">batch_size = x.size(0)
# Generate one beta per sample
beta = torch.empty(batch_size).uniform_(self.beta_lower, self.beta_higher)
softplus = ((beta * x).exp() + 1).log() / beta
res = x * torch.tanh(softplus)
</code></pre>
<p>Would that match your needs?</p>

ParticularlyPythonic: <p>I think it just might!</p>
<p>Thanks a lot!</p>
<p>I might be able to fix this fiddling around with the shapes, but I am currently getting this error:</p>
<pre><code class="lang-auto">The size of tensor a (64) must match the size of tensor b (500) at non-singleton dimension 1
</code></pre>

albanD: <p>Where does it happen?<br>
Note that I assume in the code above the x is 1D. You might need to update that.<br>
For example, if x is 2D:</p>
<ul>
<li>If you want beta to be the same for all elements for a given sample, you can simply add a <code>beta = beta.unsqueeze(-1)</code> to add a new dimension of size 1 in beta and the broadcasting logic will take care to expand it.</li>
<li>If you want a different beta for every element of every sample, then just update the size given to <code>torch.empty()</code> to reflect the full size of x.</li>
</ul>

ParticularlyPythonic: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="93550">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>beta = beta.unsqueeze(-1)</code></p>
</blockquote>
</aside>
<p>The unsqueezing fixed it.<br>
Thanks a lot!</p>

Andrews: <p>Hi , Im also trying to apply a different activation function per neuron on a simple MLP. So can you please share how you managed to separate tensors from a linear layer to apply different  activation Fs. I want my model to have seven nodes and apply different activation functions per node (first layer). I tried the select_index and the other technique mentioned but i cant do it. Im getting errors like this " ```<br>
The size of tensor a (64) must match the size of tensor b (500) at non-singleton dimension 1</p>
<pre><code class="lang-auto">"
Maybe the problems sterms from the fact that i dont how a tensor representing the layer looks like and im simply indexing it in a similar way one would index a list. Also refer me to where i can find more info about this. Thanx . sorry for the long paragraph.</code></pre>