Speed of a self-made algorithm layer in Pytorch

Leroy_Freyler: <p>Hello everyone,</p>
<p>I have a special request for the last layer of the encoder.</p>
<p>"The sum of all neurons in this layer must add up to one.</p>
<p>Example: tensor[0 -4 +5] --&gt; 0-4+5 = 1</p>
<p>For this I have called a separate function in the “forward” function, as recommended in another post in this forum, which ensures this.</p>
<p>For this I show you the called function.</p>
<pre><code class="lang-auto"># ASC function (enforce, that abundance sum is equal to 1)
def ASC_function(inputTensor):
    outputTensor = inputTensor.clone().cuda()
    for n in range(inputTensor.size(0)):
            for m in range(inputTensor.size(1)):
                # pylint: disable=E1101 # Fehler in VSC muss hier hinzugefügt werden
                outputTensor[n][m] = torch.div(inputTensor[n][m], torch.sum(inputTensor[n]).cuda()).cuda()
                # pylint: enable=E1101 # Fehler in VSC muss hier hinzugefügt werden
    return outputTensor
</code></pre>
<p>And my network.</p>
<pre><code class="lang-auto"># define the autoencoder network
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        # encoder
        self.ecf_1 = nn.Sequential(
        nn.Linear(in_features=91, out_features=9*3),
        nn.BatchNorm1d(9*3),
        nn.Linear(in_features=9*3, out_features=6*3),
        nn.BatchNorm1d(6*3),
        nn.Linear(in_features=6*3, out_features=3*3),
        nn.BatchNorm1d(3*3),
        nn.Linear(in_features=3*3, out_features=3),
        nn.BatchNorm1d(3),
        nn.LeakyReLU(300)
        )
        # encoder
        self.ecf_2 = nn.Dropout(0.5)


        # decoder
        self.dcf_1 = nn.Linear(in_features=3, out_features=91)                    
        self.dcf_2 = nn.Sigmoid()      

    def forward(self, x):
        x = self.ecf_1(x)
        x = ASC_function(x)                                       # Call the ASC (nonnegative abundances constraint) Function
        x = self.ecf_2(x)
        x = self.dcf_1(x)
        y = self.dcf_2(x)
        return y

net = Autoencoder() # runn the Auto Encoder Network from above once
</code></pre>
<p>Unfortunately this slows down the training extremely.</p>
<p>Is there a more elegant solution?</p>

ptrblck: <p>I don’t think you need the nested loop, which will slow down your code.<br>
This should also work:</p>
<pre><code class="lang-python">inputTensor = torch.randn(10, 10)
outputTensor = inputTensor.clone()
for n in range(inputTensor.size(0)):
    for m in range(inputTensor.size(1)):
        outputTensor[n][m] = torch.div(inputTensor[n][m], torch.sum(inputTensor[n]))

# without the loop
x = inputTensor.clone()
x = x / x.sum(1, keepdims=True)

print((x == outputTensor).all())
&gt; True
</code></pre>

Leroy_Freyler: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="100011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>.div(inputTensor[n][m], torch.sum(inputTensor[n]))</code></p>
</blockquote>
</aside>
<p>Hey,</p>
<p>thanks a lot for your help.</p>
<p>I tried your code. The performance has improved.</p>
<p>However, I have found that I can use the Softmax function for this just as well and the result is the same. So now I just use the Softmax function from Pytorch for my problem.</p>
<p>Many greetings</p>

ptrblck: <aside class="quote no-group" data-username="Leroy_Freyler" data-post="3" data-topic="100011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leroy_freyler/48/30056_2.png" class="avatar"> Leroy_Freyler:</div>
<blockquote>
<p>However, I have found that I can use the Softmax function for this just as well and the result is the same.</p>
</blockquote>
</aside>
<p>Are you applying <code>torch.exp</code> on the inputs before? If not, dividing by the sum alone shouldn’t yield the same output as the softmax.<br>
However, as I’m not familiar with your use case, it’s good to hear you can use built-in methods. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

Leroy_Freyler: <p>No, I had not used the exponential function before.</p>
<p>But I locked the weights for negative values and so I only needed the condition, that the row in the tensor summed must result in 1.</p>
<p>So the Softmax function is perfectly suitable.</p>