Computing the gradient and the hessian for logistic regression by hand (almost)

Ahmed_Khaled: <p>I am implementing a paper where I compute the hessian and the gradient for a logistic regression model. I am referring to those two formulas in <a href="https://www.cs.cmu.edu/~mgormley/courses/10701-f16/slides/lecture5.pdf" rel="noopener nofollow ugc">CMU ML class lectures</a>, page 57.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/574db56dfea140ebbf01f7ac6ff6a518e1db589d.png" data-download-href="https://discuss.pytorch.org/uploads/default/574db56dfea140ebbf01f7ac6ff6a518e1db589d" title="grad_hess"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/574db56dfea140ebbf01f7ac6ff6a518e1db589d_2_687x500.png" alt="grad_hess" data-base62-sha1="csk1RAUjfJ6t0Y6BGFk5Qun13Xv" width="687" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/574db56dfea140ebbf01f7ac6ff6a518e1db589d_2_687x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/7/574db56dfea140ebbf01f7ac6ff6a518e1db589d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/574db56dfea140ebbf01f7ac6ff6a518e1db589d.png 2x" data-dominant-color="A6BEBB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grad_hess</span><span class="informations">752×547 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My code is as below:</p>
<pre><code class="lang-auto">
def gradient_by_hand(loader, model, criterion, LAMBDA):
    grad = []
    for i, (data, target) in enumerate(loader):
        y_hat = model(data)
        target[target == -1] = 0
        loss = criterion(y_hat, target.to(torch.float32))
        loss.backward()
        g = model.linear.weight.grad + LAMBDA * model.linear.weight.data
        grad.append(g)
    return torch.stack(grad, dim=0).mean(dim=0)

def hessian_by_hand(loader, model, LAMBDA):
    hessian = []
    for i, (data, target) in enumerate(loader):
        y_hat = model(data)
        S = torch.diag(y_hat * (1-y_hat))
        h = torch.matmul(torch.matmul(data.T, S), data) + LAMBDA * torch.eye(data.shape[1])
        hessian.append(h)
    return torch.stack(hessian, dim=0).mean(dim=0)

</code></pre>
<p>where the model is as follows:</p>
<pre><code class="lang-auto">
class LogisticRegression(nn.Module):
    def __init__(self, input_size, num_classes):
        super(LogisticRegression, self).__init__()
        self.linear = nn.Linear(input_size, num_classes, bias= False)

    def forward(self, x):
        out = torch.squeeze(torch.sigmoid(self.linear(x)))
        return out

</code></pre>
<p>does this seem a valid implementation for the hessian and gradient of the model provided ?</p>
<p>EDIT to add some context:</p>
<p>I am performing the standard newton method to optimize the model:</p>
<pre><code class="lang-auto">def standard_newton(n_epochs, loader, model, criterion, LAMBDA, eps):

    log.info("Running standard newton method")
    for i in tqdm(range(n_epochs)):
        g = gradient_by_hand(loader, model, criterion, LAMBDA)
        h = hessian_by_hand(loader, model, LAMBDA)

        # upadte weights
        model.linear.weight.data -= torch.matmul(torch.inverse(h), torch.squeeze(g))
        # check for convergence
        if torch.norm(g) &lt; 1e-3:
            log.info("Converged because gradient norm is less than {}".format(eps))
            return model
    return model

</code></pre>