I do gradient descent manually, but something wrong

Jacksh6370: <p>Hi, I’m a noob in deep learning as well as in pytorch. The thing is I want to make a fully connnected network without using higher level api, like <em>nn.Module</em>. I’ve done that with numpy, but begin to dive deep into nn.module, I’d like to do that again in pytorch only with autograd and tensor. What I did is building a network with 3 hidden layer and 1 output layer. But something wrong when I tried to take gradient descent, because the accuracy is about 10% <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> . I don’t know what that is.<br>
I’m using mnist as dataset, and each column of <code>x_train</code> is a vector reshaped to (784, 1), so <code>x_train</code> should be (784, 500).<br>
Here’s the training code:</p>
<pre><code class="lang-python">for epoch,(x_batch,y_batch) in enumerate(train_batch):
    print(f"In training: epoch {epoch+1}, total {int(train_label.shape[0]/500)}") # 500 samples for a batch
    # Forward propagation
    print("\t"+"Forward propagation")
    f1 = self.fully_connected_layer_1_weight@x_batch + self.fully_connected_layer_1_bias
    activated1 = torch.softmax(f1/f1.sum(dim=0),dim=0)
    f2 = self.fully_connected_layer_2_weight@f1 + self.fully_connected_layer_2_bias
    activated2 = torch.softmax(f2/f2.sum(dim=0),dim=0)
    f3 = self.fully_connected_layer_3_weight@f2 + self.fully_connected_layer_3_bias
    activated3 = torch.softmax(f3/f3.sum(dim=0),dim=0)
    f4 = self.forcast_weight@f3
    activated4 = torch.softmax(f4/f4.sum(dim=0),dim=0)

    # Calculate loss
    print("\t"+"Calculate loss")
    ans = activated4[torch.argmax(input=activated4, dim=0), range(500)]
    loss = ((ans - y_batch)**2).mean()

    # Backward propagation
    print("\t"+"Backard propagation")
    loss.backward(retain_graph=True)

    # Gradient descent
    # I tried to use -= operator on weights but find -= is recorded and will be propagated
    # So I take a dirty way like this to create a new descented tensor without any computation graph.
    # And I forced self.fully_connected_layer_xs to point to descended tensors
    # But these weights don't change! 
    print("\t"+"Gradient descent")
    self.fully_connected_layer_1_weight=(\
        self.fully_connected_layer_1_weight- \
        self.fully_connected_layer_1_weight.grad*learning_rate).data.clone().detach().requires_grad_(True)
    self.fully_connected_layer_2_weight=(\
        self.fully_connected_layer_2_weight- \
        self.fully_connected_layer_2_weight.grad*learning_rate).data.clone().detach().requires_grad_(True)
    self.fully_connected_layer_3_weight=(\
        self.fully_connected_layer_3_weight- \
        self.fully_connected_layer_3_weight.grad*learning_rate).data.clone().detach().requires_grad_(True)
    self.forcast_weight=(\
        self.forcast_weight- \
        self.forcast_weight.grad*learning_rate).data.clone().detach().requires_grad_(True)
</code></pre>

albanD: <p>Hi,</p>
<p>Your gradient update is not done properly I’m afraid:</p>
<ul>
<li>You should not need <code>retain_graph=True</code>. This is only needed here because the gradient update is wrong, you can remove it.</li>
<li>To update a weight without the autograd tracking the update, you should use <code>torch.no_grad()</code>:</li>
</ul>
<pre><code class="lang-python">with torch.no_grad():
  self.fully_connected_layer_1_weight -= self.fully_connected_layer_1_weight.grad*learning_rate
</code></pre>

KFrank: <p>Hi Jack!</p>
<p>And also be sure to zero out your gradients after taking the optimization<br>
step.  (Otherwise each <code>loss.backward()</code> call will keep accumulating<br>
into the gradient.)  E.g., something like this:</p>
<pre><code class="lang-python">self.fully_connected_layer_1_weight.grad.zero_()
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Jacksh6370: <p>Thank so much! It’s exactly what I need. It works!</p>

Jacksh6370: <p>Thanks, I’ll do that.</p>