How to apply softplus on part of the output tensor

username: mishooax
cooked: <p>Here’s my simple NN structure:</p>
<pre><code class="lang-python">class DNN(nn.Module):
    def __init__(self,
                 input_layer_size: int,
                 hidden_layer_sizes: List[int],
                 dropout_rate: float,
                 debug: bool = False):
        '''
            Set up the network.
            Args:
                input_layer_size: size of the input layer
                hidden_layer_sizes: sizes of the hidden linear layers
                    e.g. [5,5,3,2] -&gt; linear layers 5,5 -&gt; 5,3 -&gt; 3,2
                dropout_rate: dropout rate
        '''
        super().__init__()

        self.debug = debug
        self.linear_layers_list: List[nn.Module] = []

        self.linear_layers.append(nn.Linear(input_layer_size, hidden_layer_sizes[0]))
        self.linear_layers.append(nn.LeakyReLU(0.04))
        self.linear_layers.append(nn.BatchNorm1d(hidden_layer_sizes[0]))
        self.linear_layers.append(nn.Dropout(p=dropout_rate))

        # hidden layers
        for in_size, out_size in zip(hidden_layer_sizes[:-2], (hidden_layer_sizes[1:-1])):
            self.linear_layers.append(nn.Linear(in_size, out_size))
            self.linear_layers.append(nn.LeakyReLU(0.04))
            self.linear_layers.append(nn.BatchNorm1d(out_size))
            self.linear_layers.append(nn.Dropout(p=dropout_rate))

        # output layer
        output_layer_mean = nn.Linear(hidden_layer_sizes[-1], 1)
        output_layer_sigma = nn.Linear(hidden_layer_sizes[-1], 1)
        # make sure that sigma &gt; 0
        output_layer = torch.cat(output_layer_mean, F.softplus(output_layer_sigma))  &lt;--- NO GOOD
        self.linear_layers_list.append(output_layer)

        self.linear_layers = torch.nn.ModuleList(self.linear_layers_list)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        for layer in self.linear_layers_list:
            x = layer(x)
        return x
</code></pre>
<p>I need the output of <code>output_layer_sigma</code> to be positive so I’m running it through a <code>softplus</code>. However, I get the following error at run time:</p>
<pre><code class="lang-auto">File "(...)", line 56, in __init__
    output_layer = torch.cat(output_layer_mean, F.softplus(output_layer_sigma)) 
TypeError: softplus(): argument 'input' (position 1) must be Tensor, not Linear
</code></pre>
<p>How do I make sure the 2nd value in my output tensor is always positive? should i call <code>F.softplus</code> in <code>forward</code> instead of <code>__init__()</code>? Or use the <code>nn.Softplus</code> module? Or is there a better, “pytorch-standard” way of doing things? Thanks!</p>
<p>PS: Here’s what this would look like in <code>tf.keras</code>:</p>
<pre><code class="lang-python"># output layer - enforce a positive sigma
mean_output = keras.layers.Dense(1)(last_hidden_layer)
sigma_output = keras.layers.Dense(1, activation='softplus')(last_hidden_layer)
output_layer = keras.layers.concatenate([mean_output, sigma_output])
</code></pre>

username: KFrank
cooked: <p>Hello Mishoo!</p>
<aside class="quote no-group quote-modified" data-username="mishooax" data-post="1" data-topic="100047" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mishooax/48/46230_2.png" class="avatar"> mishooax:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">class DNN(nn.Module):
       ...
        # output layer
        output_layer_mean = nn.Linear(hidden_layer_sizes[-1], 1)
        output_layer_sigma = nn.Linear(hidden_layer_sizes[-1], 1)
        # make sure that sigma &gt; 0
        output_layer = torch.cat(output_layer_mean, F.softplus(output_layer_sigma))  &lt;--- NO GOOD
        self.linear_layers_list.append(output_layer)
...
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        for layer in self.linear_layers_list:
            x = layer(x)
        return x
</code></pre>
<p>should i call <code>F.softplus</code> in <code>forward</code> instead of <code>__init__()</code>?</p>
</blockquote>
</aside>
<p>Yes.  As you see, you can’t apply <code>softplus()</code> to a <code>Linear</code>.  You need<br>
to apply it to the output of the <code>Linear</code>, which is a tensor.</p>
<p>I would not append <code>output_layer</code> (nor <code>output_layer_mean</code> nor<br>
<code>output_layer_sigma</code>) to <code>linear_layers_list</code>.  Something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">        output_layer_mean = nn.Linear(hidden_layer_sizes[-1], 1)
        output_layer_sigma = nn.Linear(hidden_layer_sizes[-1], 1)
        # do this stuff in forward ...
        # # make sure that sigma &gt; 0
        # output_layer = torch.cat(output_layer_mean, F.softplus(output_layer_sigma))  &lt;--- NO GOOD
        # self.linear_layers_list.append(output_layer)
...
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        for layer in self.linear_layers_list:
            x = layer(x)
        # haven't yet applied output_layer_mean nor output_layer_sigma 
        x = torch.cat((output_layer_mean(x), F.softplus(output_layer_sigma(x))), dim = -1)
        return x
</code></pre>
<p>Best.</p>
<p>K. Frank</p>