What does the backward() function do?

somghosh: <p>I have two networks, “net1” and "net2"<br>
Let us say “loss1” and “loss2” represents the loss function of “net1” and “net2” classifier’s loss.<br>
lets say “optimizer1” and “optimizer2” are the optimizers of both networks.</p>
<p>“net2” is a pretrained network and I want to backprop the (gradients of) the loss of “net2” into “net1”.<br>
loss1=…some loss defined<br>
So, loss1 = loss1 + loss2    (lets say that loss2 was defined initially)</p>
<p>So I do<br>
loss1.backward(retain_graph=True)        <span class="hashtag">#what</span> happens when I write this<br>
loss2.backward()<br>
optimizer2.step()</p>
<p>What is the difference between backward() and step() ???</p>
<p>If I do not write loss1.backward() what will happen ??</p>

colesbury: <p><code>loss.backward()</code> computes <code>dloss/dx</code> for every parameter <code>x</code> which has <code>requires_grad=True</code>. These are accumulated into <code>x.grad</code> for every parameter <code>x</code>. In pseudo-code:</p>
<pre><code class="lang-auto">x.grad += dloss/dx
</code></pre>
<p><code>optimizer.step</code> updates the value of <code>x</code> using the gradient <code>x.grad</code>. For example, the SGD optimizer performs:</p>
<pre><code class="lang-auto">x += -lr * x.grad
</code></pre>
<p><code>optimizer.zero_grad()</code> clears <code>x.grad</code> for every parameter <code>x</code> in the optimizer. It’s important to call this before <code>loss.backward()</code>, otherwise you’ll accumulate the gradients from multiple passes.</p>
<p>If you have multiple losses (loss1, loss2) you can sum them and then call backwards once:</p>
<pre><code class="lang-auto">loss3 = loss1 + loss2
loss3.backward()
</code></pre>

somghosh: <p>Hi <a class="mention" href="/u/colesbury">@colesbury</a>, thanks for your illustration.</p>
<p>I have one more question. Lets say I want to backprop “loss3” into “net1” and do not want to backprop “loss 2” to “net2”. In that case I should not write</p>
<blockquote>
<p>loss2.backward()<br>
optimizer2.step().</p>
</blockquote>
<p>I should only write</p>
<blockquote>
<p>loss3 = loss1 + loss2<br>
loss3.backward().             RIGHT ??</p>
</blockquote>
<p>In case I have written loss2.backward() and have not written  optimizer2.step(), will that affect my gradients when I compute loss3.backward(). ???</p>

sarthak268: <p>Does backward update the weights, if we do not use optimizer?</p>

SimonW: <p>no it just computes gradients</p>

John1231983: <p>Very clear explaination!. If I have two losses:loss1 get data from dataloader1 with batch size of 4, while loss2 get data from loader2 with batch size of 4. Then what is batch size of loss=loss1+loss2? Is it 4 or 8? The code likes</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss1=CrossEntropyLoss(output1,target1)
loss2=CrossEntropyLoss(output2,target2)
loss=loss1+loss2
loss.backward()
optimizer.step()

</code></pre>

aerinykim: <p>No, it does not. Update happens only when you call step().</p>

Brando_Miranda: <p>I was reading <a href="https://jdhao.github.io/2017/11/12/pytorch-computation-graph/" rel="noopener nofollow ugc">here</a> that:</p>
<blockquote>
<p>In the next iteration, a fresh new graph is created and ready for back-propagation.</p>
</blockquote>
<p>I am wondering. When <strong>exactly</strong> is the fresh new graph created? Is it when we call:</p>
<ul>
<li><code>optimizer.zero_grad()</code></li>
<li><code>output.backward()</code></li>
<li><code>optimizer.step()</code></li>
</ul>
<p>or at some other time?</p>

colesbury: <blockquote>
<p>I am wondering. When  <strong>exactly</strong>  is the fresh new graph created?</p>
</blockquote>
<p>It’s created during the forward pass. i.e. when you write something like:</p>
<p><code>loss = criterion(model(input), target)</code></p>
<p>The graph is accessible through <code>loss.grad_fn</code> and the chain of autograd <code>Function</code> objects.</p>
<p>The graph is <strong>used</strong> by <code>loss.backward()</code> to compute gradients.</p>
<p><code>optimizer.zero_grad()</code> and <code>optimizer.step()</code> do not affect the graph of autograd objects. They only touch the model’s parameters and the parameter’s <code>grad</code> attributes.</p>

Oli: <p>If there are several branches / subgraphs - would it be beneficial or even possible to do loss.backward() on the subgraphs? I’m hoping that when one branch finishes early it might free up memory this way.</p>
<p>I know that you can add the losses together and do one losses.backward() btw</p>

psu1: <p>Here,  x only represent the parameters that contribute to the ‘loss’,  right ? (a.k.a  loss.backward() doesn’t affect other variables that without contribution to ‘loss’ )</p>

ttoosi: <p>Hi,</p>
<p>I want to implement the backward graph separately which means dropping <code>loss.backward()</code> and substituting that with a network that accepts error as input and gives gradients in each layer. For example, for MSE loss it is intuitive to use <code>error = target-output</code> as the input to the backward graph (which is in fully_connected network, is the transposed of the forward graph).<br>
Pytorch loss functions give the loss and not the tensor which is given as input to the backward graph. Is there any easy way to access the input to the backward graph after computing loss? (e.g. <code>loss = nn.CrossEntropyLoss()(outputs, targets)</code>)</p>
<p>Thanks</p>

ptrblck: <p>I’m not sure which input you are looking for, but you can pass the <code>gradient</code> directly to the <code>backward</code> function.<br>
The default would be a scalar value of 1. If you need to provide a specific gradient, you could use <code>loss.backward(gradient=...)</code>.</p>

ttoosi: <p>Thank you for your response.<br>
I explained my confusion with loss.backward() in <a href="https://discuss.pytorch.org/t/access-to-grad-output-the-input-to-the-backward/57349/5">another topic</a>.</p>

Mina_soltan: <p>I have a very similar implementation however it askes for retain_graph=True and then that slows down the code so that it’s impractical to train. Any thoughts?</p>

Basavaraj_Patil: <p>why the x.grad is added with the dloss/dx… Insted it should be multiplied with learning rate and add with the older weight values.<br>
Because, new_weight = old_weight - (learning_rate)*x.grade</p>

Basavaraj_Patil: <ol>
<li>When we pass the 1st batch for the forward pass and compute the loss for the 1st batch.</li>
<li>We calculate the back propagation to compute d_loss/dx for all layers.</li>
<li>Then with optimization technique we updates the weights with help of optimizer.step function for the 1st batch.<br>
Later for the second batch whether the updated weights from the 1st batch  will be used or what. And before applying the backward() function for second batch weather we should do optimizer.zero_grade() or WHAT???</li>
</ol>

saba: <p>Hi <a class="mention" href="/u/colesbury">@colesbury</a></p>
<p>I used two loss function loss=loss1+loss2, and I expect to have different gradient when I use just loss=loss1,But the gradient flow and numbers was same.indeed adding second loss does not have any effect. Would you pleas help me with that? I try different second loss but the result does not have any change. The first loss is BCELoss and the second one is L1. I change the sigmoid function to Relu, But again the gradient from backward.() with loss2 and without loss2 is same!</p>
<pre><code class="lang-auto">netG = Generator(ngpu,nz,ngf).to(device)

optimizerG = optim.Adam(netG.parameters(), lr=lr2, betas=(beta1, 0.999))

netG.zero_grad()

label.fill_(real_label)  
label=label.to(device)
output = netD(fake).view(-1)
# Calculate G's loss based on this output
loss1 = criterion(output, label)


xxx=torch.histc(Gaussy.squeeze(1).view(-1).cpu(),100, min=0, max=1, out=None)
ddGaussy=xxx/xxx.sum()

xxx1=torch.histc(fake.squeeze(1).view(-1).cpu(),100, min=0, max=1, out=None)
ddFake=xxx1/xxx1.sum()
loss2=abs(ddGaussy-ddFake).sum()

# Calculate gradients for G with 2 loss

errG=loss1+loss2
errG.backward()

for param in netG.parameters():
            print(param.grad.data.sum())

# Update G
optimizerG.step()




 
## ------------------
class Generator(nn.Module):
    def __init__(self,ngpu,nz,ngf):
        super(Generator, self).__init__()
        self.ngpu=ngpu
        self.nz=nz
        self.ngf=ngf
        self.l1= nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d(self.nz, self.ngf * 8, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 8),
            nn.ReLU(True),)
            # state size. (ngf*8) x 4 x 4
        self.l2=nn.Sequential(nn.ConvTranspose2d(self.ngf * 8, self.ngf * 4, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 4),
            nn.ReLU(True),)
            # state size. (ngf*4) x 8 x 8
        self.l3=nn.Sequential(nn.ConvTranspose2d( self.ngf * 4, self.ngf * 2, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 2),
            nn.ReLU(True),)
            # state size. (ngf*2) x 16 x 16
        self.l4=nn.Sequential(nn.ConvTranspose2d( self.ngf*2, 1, 3, 1, 0, bias=False),nn.Sigmoid()
#            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        out=self.l1(input)
        out=self.l2(out)
        out=self.l3(out)
        out=self.l4(out)
        print(out.shape)
        return out</code></pre>

ptrblck: <p>Double post with answer from <a href="https://discuss.pytorch.org/t/why-by-adding-second-loss-and-applying-the-back-ward-generated-gradient-is-as-same-as-before/94625/2">here</a>.</p>

Padmaksha_Roy: <p>hi <a class="mention" href="/u/colesbury">@colesbury</a>, I am trying to do a similar thing where I have a reconstruction loss and a kernel alignment loss. They are calculated as below:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.We1 = torch.nn.Parameter(torch.Tensor(input_length, args.hidden_size).uniform_(-1.0 / math.sqrt(input_length), 1.0 / math.sqrt(input_length)))
        self.We2 = torch.nn.Parameter(torch.Tensor(args.hidden_size, args.code_size).uniform_(-1.0 / math.sqrt(args.hidden_size), 1.0 / math.sqrt(args.hidden_size)))

        self.be1 = torch.nn.Parameter(torch.zeros([args.hidden_size]))
        self.be2 = torch.nn.Parameter(torch.zeros([args.code_size]))

    def encoder(self, encoder_inputs):
        hidden_1 = torch.tanh(torch.matmul(encoder_inputs.float(), self.We1) + self.be1)
        code = torch.tanh(torch.matmul(hidden_1, self.We2) + self.be2)
        return code

    def decoder(self,encoder_inputs):
        code = self.encoder(encoder_inputs)

        # ----- DECODER -----
        if tied_weights:

            Wd1 = torch.transpose(We2)
            Wd2 = torch.transpose(We1)

        else:

            Wd1 = torch.nn.Parameter(
                torch.Tensor(args.code_size, args.hidden_size).uniform_(-1.0 / math.sqrt(args.code_size),
                                                                           1.0 / math.sqrt(args.code_size)))
            Wd2 = torch.nn.Parameter(
                torch.Tensor(args.hidden_size, input_length).uniform_(-1.0 / math.sqrt(args.hidden_size),
                                                                             1.0 / math.sqrt(args.hidden_size)))

            bd1 = torch.nn.Parameter(torch.zeros([args.hidden_size]))
            bd2 = torch.nn.Parameter(torch.zeros([input_length]))

            if lin_dec:
                hidden_2 = torch.matmul(code, Wd1) + bd1
            else:
                hidden_2 = torch.tanh(torch.matmul(code, Wd1) + bd1)

            dec_out = torch.matmul(hidden_2, Wd2) + bd2

        return  dec_out

    def kernel_loss(self,code, prior_K):
        # kernel on codes
        code_K = torch.mm(code, torch.t(code))

        # ----- LOSS -----
        # kernel alignment loss with normalized Frobenius norm
        code_K_norm = code_K / torch.linalg.matrix_norm(code_K, ord='fro', dim=(- 2, - 1))
        prior_K_norm = prior_K / torch.linalg.matrix_norm(prior_K, ord='fro', dim=(- 2, - 1))
        k_loss = torch.linalg.matrix_norm(torch.sub(code_K_norm,prior_K_norm), ord='fro', dim=(- 2, - 1))
        return k_loss

# Initialize model
model = Model()
</code></pre>
<p>Now, during training I pass my training data as inputs to the encoder and decoder.</p>
<pre><code class="lang-auto">for ep in range(args.num_epochs):
    for batch in range(max_batches):
        # get input data
            
            dec_out = model.decoder(encoder_inputs)
            reconstruct_loss = torch.mean((dec_out - encoder_inputs) ** 2)
            enc_out = model.encoder(encoder_inputs)
            k_loss = model.kernel_loss(enc_out,prior_K)
       

            tot_loss = reconstruct_loss + args.w_reg * reg_loss + args.a_reg * k_loss
            tot_loss = tot_loss.float()

            # Backpropagation
            optimizer.zero_grad()
            #tot_loss.backward(retain_graph=True)
            tot_loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_gradient_norm)
            optimizer.step()

</code></pre>
<p>This always gives me an error saying “RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed).<br>
Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need<br>
to backward through the graph a second time”</p>
<p>It works only when I activate the retain_graph flag. But takes huge time for training. Can you please let me know what wrong I am doing here?</p>
<p>Thank you!</p>