Are torch.nn.ReLU and torch.nn.Sigmoid trainable?

username: evilroach
cooked: <p>I build a simple GRU model with PyTorch. It includes 4 sub-modules.</p>
<p>I noted that some dictionaries return by the <code>state_dict()</code> of them are empty after training, while ones of the other sub-modules certainly have some weights and bias.</p>
<p>The code:</p>
<pre><code class="lang-auto">class GruModel(nn.Module):
    def __init__(self, inputs, nodes, layers=2):
        super(GruModel, self).__init__()
        self.gru_m = nn.GRU(input_size=inputs, num_layers=layers, hidden_size=nodes,
                            batch_first=True, dropout=0.5)
        self.activt_f = nn.ReLU()
        self.output_f = nn.Linear(nodes, 1)
        self.probab_f = nn.Sigmoid()

    def forward(self, x, h):
        o, h = self.gru_m(x, h)
        o = self.activt_f(o[:, -1])
        out = self.output_f(o)
        return self.probab_f(out)

    def trainWith(self, ...):
        ''' training body '''
        criterion = nn.BCEWithLogitsLoss()
        optimizer = torch.optim.Adadelta(self.parameters(), lr=learn_rat)
        lr_schdlr = torch.optim.lr_scheduler.ExponentialLR(
            optimizer, gamma=0.99)
        t_loader = torch_data.DataLoader(...)
        for e in range(epochs):
            for x, p_label, n_label in t_loader:
                optimizer.zero_grad()
                out = self(x, self.init_hidden(batch_size))
                loss = criterion(out, p_label)
                loss.backward()
                optimizer.step()
            lr_schdlr.step()

    def save(self, full_path: str):
        print(self.gru_m.state_dict())
        print(self.activt_f.state_dict())
        print(self.output_f.state_dict())
        print(self.probab_f.state_dict())
</code></pre>
<p>In real running, the state_dict of the sub-module <code>self.gru_m</code> and <code>self.output_f</code> have values as expected, but the ones of the sub-module <code>self.activt_f</code> (nn.ReLU) and <code>self.probab_f</code> (nn.Sigmoid) have nothing.</p>
<p>Don’t care about my training process, I feed it with tons of data and run through hundreds of epochs, and the model can do classification as I expected.</p>
<p>I’m interested in whether the later two modules are trainable, or they do <strong>NOT</strong> need any weights and bias with them to do their work?</p>
<p>If so, can we say that the <code>torch.nn.Sigmoid</code> is same as <code>torch.nn.functional.sigmoid</code>? Because they are all dummy functions, not stateful objects.</p>
<p>Thanks a lot!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="evilroach" data-post="1" data-topic="162243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evilroach/48/53515_2.png" class="avatar"> evilroach:</div>
<blockquote>
<p>If so, can we say that the <code>torch.nn.Sigmoid</code> is same as <code>torch.nn.functional.sigmoid</code>? Because they are all dummy functions, not stateful objects.</p>
</blockquote>
</aside>
<p>Yes, <code>nn.Sigmoid</code> is a module wrapper around the stateless <code>torch.sigmoid</code> and <code>nn.ReLU</code> the equivalent wrapper for the stateless <code>torch.relu</code>. Neither of these modules has trainable parameters.</p>

username: evilroach
cooked: <p>Thank you very much!</p>
<p>Further more, if I use torch.Sigmoid(or torch.nn.functional.sigmoid) instead of nn.Sigmoid, can I get a little benifit of performance?</p>

username: ptrblck
cooked: <p>No, I would not expect to see any significant difference in “real” workloads between using a module vs. its functional API.<br>
“Real” workloads refers to training or inference scripts which are not profiling these calls in isolation.</p>