Custom Sigmoid (the network is not trained)

boomland: <p>I tried to write my custom layer for sigmoid. But it doesn’t work. The network is not trained. If I replace the activation function with the standard torch.nn.Sigmoid() then it starts learning and achieves great accuracy. I can’t figure out what’s wrong with my custom sigmoid <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:"> Help me please, guys! Thanks</p>
<p>Sigmoid custom module:</p>
<pre><code class="lang-auto">class SigmoidCustom(torch.nn.Module):
    def __init__(self):
        super(SigmoidCustom, self).__init__()
    
    def forward(self, x):
        return 1. / (1 + torch.exp(-x))
</code></pre>
<p>Network:</p>
<pre><code class="lang-auto">class LinearNet(torch.nn.Module):
    def __init__(self, in_features,  out_features, hid_neurons=200):
        super(LinearNet, self).__init__()
        self.lin1 = LinearCustom(in_features, hid_neurons, bias=True)
        self.act1 = SigmoidCustom()
        self.lin2 = LinearCustom(hid_neurons, hid_neurons, bias=True)
        self.act2 = SigmoidCustom()
        self.lin3 = LinearCustom(hid_neurons, out_features, bias=True)
        self.softmax = torch.nn.Softmax(dim=1)
    
    def forward(self, x):
        # x - bs x in_feat
        x = self.lin1(x) # bs x hid_neurons
        x = self.act1(x) # bs x hid_neurons
        x = self.lin2(x) # bs x hid_neurons
        x = self.act2(x) # bs x hid_neurons
        x = self.lin3(x) # bs x out_feat
        return x

    def inference(self, x):
        x = self.softmax(x)
        return x
</code></pre>

KFrank: <p>Hi Yaroslav!</p>
<aside class="quote no-group" data-username="boomland" data-post="1" data-topic="142250" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boomland/48/45980_2.png" class="avatar"> boomland:</div>
<blockquote>
<p>I tried to write my custom layer for sigmoid. But it doesn’t work.</p>
</blockquote>
</aside>
<p>Your <code>SigmoidCustom</code> works for me:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; class SigmoidCustom(torch.nn.Module):
...     def __init__(self):
...         super(SigmoidCustom, self).__init__()
...
...     def forward(self, x):
...         return 1. / (1 + torch.exp(-x))
...
&gt;&gt;&gt; s = torch.randn (5)
&gt;&gt;&gt; t = s.clone()
&gt;&gt;&gt; s.requires_grad = True
&gt;&gt;&gt; t.requires_grad = True
&gt;&gt;&gt; s.sigmoid().sum().backward()
&gt;&gt;&gt; s.grad
tensor([0.2477, 0.2433, 0.2467, 0.2061, 0.2474])
&gt;&gt;&gt; SigmoidCustom() (t).sum().backward()
&gt;&gt;&gt; t.grad
tensor([0.2477, 0.2433, 0.2467, 0.2061, 0.2474])
&gt;&gt;&gt; torch.allclose (s.grad, t.grad)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

boomland: <p>Hi Frank!<br>
Thank you for your answer! Yes, I also checked that the result of my module is the same as the library. But for some reason, with my sigmoid module, the network is not trained. I still can’t figure out why <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:"> maybe in torch.nn.Sigmoid somehow works with gradients in a special way… idk</p>

boomland: <p>here is the whole network training code:</p>
<pre><code class="lang-auto">train_data = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=None)
test_data = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=None)

X_train, y_train = train_data.data, train_data.targets
X_test, y_test = test_data.data, test_data.targets
X_train = X_train.float()
X_test = X_test.float()
X_train = X_train.reshape(-1, 28 * 28)
X_test = X_test.reshape(-1, 28 * 28)

device = 'cuda:0' if torch.cuda.is_available() else 'cpu'

in_feat = 28 * 28
out_feat = 10
net = LinearNet(in_feat, out_feat, hid_neurons=25)
net = net.to(device)

optimizer = torch.optim.Adam(net.parameters(), lr=3e-4)
criterion = torch.nn.CrossEntropyLoss()

num_epoch = 500
batch_size = 100

history_loss = {'train': [], 'test': []}
history_acc = {'train': [], 'test': []}

def generate_batch(X, y, batch_size):
    idx = torch.randperm(len(X))[:batch_size]
    return X[idx], y[idx]

iter_per_epoch = int(np.ceil(len(X_train) / batch_size))

X_test, y_test = X_test.to(device), y_test.to(device)

for epoch in range(num_epoch):

    total_epoch_loss = 0.0
    total_epoch_acc = 0.0
    for iter in range(iter_per_epoch):
        X_batch, y_batch = generate_batch(X_train, y_train, batch_size)
        X_batch = X_batch.to(device)
        y_batch = y_batch.to(device)

        output = net(X_batch)
        pred = torch.argmax(output, dim=1)
        batch_acc = (pred == y_batch).float().mean()
        total_epoch_acc += batch_acc

        loss = criterion(output, y_batch)
        total_epoch_loss += loss

        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

    total_epoch_acc /= iter_per_epoch
    total_epoch_loss /= iter_per_epoch
    history_acc['train'].append(total_epoch_acc)
    history_loss['train'].append(total_epoch_loss)

    # validation
    with torch.no_grad():
        output = net(X_test)
        pred = torch.argmax(output, dim=1)
        acc = (pred == y_test).float().mean()
        loss = criterion(output, y_test)
        history_acc['test'].append(acc)
        history_loss['test'].append(loss)

        if epoch % 10 == 9: 
            print('Epoch {}, validation acc = {:.4f}'.format(epoch+1, acc))
</code></pre>

KFrank: <p>Hi Yaroslav!</p>
<p>Here’s a little more information:</p>
<p>It turns out the the gradient for <code>torch.sigmoid()</code> is somewhat worse<br>
than it could be.  Your <code>SigmoidCustom</code> displays similar imperfect<br>
behavior, but is rather better than <code>torch.sigmoid()</code>.</p>
<p>See this post:</p>
<aside class="quote quote-modified" data-post="1" data-topic="142352">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/imperfect-gradient-for-torch-sigmoid/142352">Imperfect gradient for torch.sigmoid()</a> 
  </div>
  <blockquote>
    Hi All! 
The gradient computation for torch.sigmoid() is not as good as it 
could or should be. 
Specifically, it underflows to zero sooner than it should. 
This can be seen by direct inspection of the gradient values.  Also, 
the gradient should be symmetric around zero, and this condition 
is violated.  Notably, the gradient underflows early for positive values 
of x. 
Also, the standard formula, sigmoid (x)  =  1 / (1 + exp (-x)), 
performs better, although still imperfectly, in that its grad…
  </blockquote>
</aside>

<p>Nonetheless, I think it is unlikely that this difference between your<br>
<code>SigmoidCustom</code> and pytorch’s standard implementation explains<br>
the issue you are seeing.  Both versions of <code>sigmoid()</code> are basically<br>
okay, and the imperfection in their gradients is more of an edge case.</p>
<p>I would only think this would matter if your network were close to<br>
training unstably.</p>
<p>I would suggest trying something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">class LinearNet(torch.nn.Module):
    def __init__(self, in_features,  out_features, hid_neurons=200, useCustom = True):
        super(LinearNet, self).__init__()
        if  useCustom:
            sig = SigmoidCustom()
        else:
            sig = torch.nn.Sigmoid()
        self.lin1 = LinearCustom(in_features, hid_neurons, bias=True)
        self.act1 = sig
        self.lin2 = LinearCustom(hid_neurons, hid_neurons, bias=True)
        self.act2 = sig
        self.lin3 = LinearCustom(hid_neurons, out_features, bias=True)
        self.softmax = torch.nn.Softmax(dim=1)
</code></pre>
<p>This way you can switch between the standard and custom <code>sigmoid()</code><br>
with a single flag and not risk letting some other bug creep in by changing<br>
additional code.</p>
<p>Best.</p>
<p>K. Frank</p>