How to add a function between layers?

Konstantin_Lebeda: <p>Hello! How can I add functions, different for every neuron, between two layers of fully connected NN? Is it possible?</p>
<pre><code>class PolyNet(torch.nn.Module):

    def __init__(self):
        super(PolyNet, self).__init__()
        self.fc1 = torch.nn.Linear(1, 4, bias=False)
        # something here?
        self.fc2 = torch.nn.Linear(4, 1, bias=False)

    def forward(self, x):
        x = self.fc1(x)
        # or mb here?
        x = self.fc2(x)

    return x
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a1bc2626345ec38576ebe4030db9a71d1d3fbb9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3a1bc2626345ec38576ebe4030db9a71d1d3fbb9" title="forum1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a1bc2626345ec38576ebe4030db9a71d1d3fbb9_2_690x452.jpeg" alt="forum1" data-base62-sha1="8i3angaAk5LtEe7aW5fXDowSlwl" width="690" height="452" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a1bc2626345ec38576ebe4030db9a71d1d3fbb9_2_690x452.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a1bc2626345ec38576ebe4030db9a71d1d3fbb9.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a1bc2626345ec38576ebe4030db9a71d1d3fbb9.jpeg 2x" data-dominant-color="F9FCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">forum1</span><span class="informations">853×560 58.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Yes, you can freely manipulate the intermediate activation <code>x</code> (output of <code>fc1</code>) in your <code>forward</code> method before passing it to the next layer.</p>

Konstantin_Lebeda: <p>Thanks, but how to apply function to specific neuron? Would you mind to give me example in code please</p>

ptrblck: <p>I assume “neuron” means a specific output activation in this use case.<br>
If so, you can directly apply the method on it and concatenate the results afterwards.<br>
Depending on your actual use case, something like this would work:</p>
<pre><code class="lang-python">class PolyNet(torch.nn.Module):
    def __init__(self):
        super(PolyNet, self).__init__()
        self.fc1 = torch.nn.Linear(1, 4, bias=False)
        self.fc2 = torch.nn.Linear(4, 1, bias=False)

    def forward(self, x):
        x = self.fc1(x)
        print("DEBUG: before\n{}".format(x))
        # apply method on "neuron2"
        x_tmp = x[:, 2] # index activation
        x_tmp = x_tmp * 1000. # apply op on neuron
        x = torch.cat((x[:, :2], x_tmp[:, None], x[:, 3:]), dim=1) # concatenate
        print("DEBUG: after\n{}".format(x))
        x = self.fc2(x)
        return x
    
model = PolyNet()
x = torch.randn(2, 1)
out = model(x)
# DEBUG: before
# tensor([[ 0.6320, -0.1368,  0.3954,  0.0055],
#         [-0.3955,  0.0856, -0.2474, -0.0034]], grad_fn=&lt;MmBackward0&gt;)
# DEBUG: after
# tensor([[ 6.3205e-01, -1.3676e-01,  3.9545e+02,  5.4939e-03],
#         [-3.9545e-01,  8.5570e-02, -2.4742e+02, -3.4374e-03]],
#        grad_fn=&lt;CatBackward0&gt;)
</code></pre>