Copy model's gradients are none

Shuvo_Saha_Roy: <p>Hey, I was trying to implement sarah optimization algorithm.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c7a5bce5173a190eed864c65f467fe2056fdab6.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c7a5bce5173a190eed864c65f467fe2056fdab6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c7a5bce5173a190eed864c65f467fe2056fdab6_2_501x500.png" alt="image" data-base62-sha1="hLbmdVRvNK2dd1PtPbU5jjxeubk" width="501" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c7a5bce5173a190eed864c65f467fe2056fdab6_2_501x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c7a5bce5173a190eed864c65f467fe2056fdab6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c7a5bce5173a190eed864c65f467fe2056fdab6.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">703×701 61.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here is the code.</p>
<h1><a name="h-1-model-1" class="anchor" href="#h-1-model-1"></a>1) Model</h1>
<h1><a name="linear-model-f-wx-b-sigmoid-at-the-end-2" class="anchor" href="#linear-model-f-wx-b-sigmoid-at-the-end-2"></a>Linear model f = wx + b , sigmoid at the end</h1>
<p>class Model(nn.Module):<br>
def <strong>init</strong>(self, n_input_features):<br>
super(Model, self).<strong>init</strong>()<br>
self.linear = nn.Linear(n_input_features, 1)</p>
<pre><code>def forward(self, x):
    y_pred = torch.sigmoid(self.linear(x))
    return y_pred
</code></pre>
<p>model = Model(n_features)</p>
<p>def binary_cross_entropy_loss(outputs, targets):<br>
epsilon = 1e-4  # small epsilon value to ensure numerical stability<br>
loss = - (targets * torch.log(outputs + epsilon) + (1 - targets) * torch.log(1 - outputs + epsilon))<br>
return torch.mean(loss)</p>
<h1><a name="h-2-loss-and-optimizer-3" class="anchor" href="#h-2-loss-and-optimizer-3"></a>2) Loss and optimizer</h1>
<p>num_epochs = 100<br>
learning_rate = 0.01<br>
criterion = binary_cross_entropy_loss<br>
optimizer = torch.optim.Adagrad(model.parameters(), lr=learning_rate)<br>
train_losses = np.zeros(num_epochs)</p>
<h1><a name="h-3-training-loop-4" class="anchor" href="#h-3-training-loop-4"></a>3) Training loop</h1>
<p>for epoch in range(num_epochs):<br>
# Forward pass and loss<br>
y_pred = model(X_train)<br>
loss = criterion(y_pred, y_train)</p>
<pre><code>old_model = copy.deepcopy(model)

optimizer.zero_grad()  # Clear previous gradients
loss.backward(retain_graph=True)  # Compute gradients using backward pass
# Access gradients and store them in a variable
gradients = [param.grad for param in model.parameters()]

V0 = [torch.zeros_like(p) for p in model.parameters()]

for i, w in enumerate(model.parameters()):
    V0[i] +=  gradients[i]
    w.data = w.data - V0[i] * learning_rate

m = 2*num_epochs
inside_loop = np.zeros(m)
n = len(X_train)
for t in range(0, m):
    # Sample a random index uniformly at random from [n]
    i_t = random.randint(0, n - 1)
    # Access the sample at index i_t
    sample = X_train[i_t]
    target = y_train[i_t]

    ### One sample and it's gradient with updated model
    sample_pred = model(sample)
    inner_loss = criterion(sample_pred, target)

    # Backward pass
    optimizer.zero_grad()  # Clear previous gradients
    inner_loss.backward(retain_graph=True)  # Compute gradients using backward pass
    # Access gradients and store them in a variable
    inner_gradients_wrt_updated_model = [param.grad for param in model.parameters()]

    ### One sample and it's gradient with old model
    sample_pred_old = old_model(sample)
    inner_loss_old = criterion(sample_pred, target)
    # Backward pass
    optimizer.zero_grad()  # Clear previous gradients 
    inner_loss_old.backward(retain_graph=True)  # Compute gradients using backward pass
    # Access gradients and store them in a variable
    inner_gradients_wrt_old_model = [param.grad for param in old_model.parameters()]


    for c,d in enumerate(model.parameters()):
        V0[c] = inner_gradients_wrt_updated_model[c] - inner_gradients_wrt_old_model[c] + V0[c]
        d.data = d.data - learning_rate * V0[c]

    inside_loop[t] = loss.item()


# zero grad before new step
optimizer.zero_grad()

train_losses[epoch] = loss.item()

if (epoch+1) % 10 == 0:
    print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')
</code></pre>
<p>In this code inner_gradients_wrt_old_model always is None. And I failed to solve it. If you can help me regrading this I will be grateful to you.</p>
<p>I really appreciate any help you can provide.</p>
<p>And let me know if the code match with algorithm implementation or not.</p>

AidanGoldfarb: <p>I have seen some things <a href="https://github.com/pytorch/tutorials/issues/2177" rel="noopener nofollow ugc">git issue</a>, <a href="https://github.com/pytorch/tutorials/issues/2177" rel="noopener nofollow ugc">forum post</a> and <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">docs</a> which suggest that perhaps copying the model.state_dict() separately could solve this. Not sure if deepcopy (from the first hyperlink) was updated to fix the issue.</p>