How can the variable computed in the previous epoch be used for computing the loss function in the next epoch?

ddguoll: <p>The code is given below. How can the  variable computed in the previous epoch be used for<br>
computing the loss function in the next epoch?</p>
<p>for epoch in range((args.start_epoch+1), args.epochs):<br>
for input, target in train_loader:<br>
target = target.cuda()<br>
input=input.cuda()<br>
input_var = torch.autograd.Variable(input)<br>
target_var = torch.autograd.Variable(target)<br>
outputs, feature = model(input_var)<br>
if epoch&gt;0:<br>
l= criterion.forward(feature,target_var, Fea)<br>
Fea=function(model,train_loader)</p>

ptrblck: <p>If you want to use <code>feature</code> from the previous iteration, you could store it in another variable so that it won’t be overwritten by the <code>model(input_var)</code> call.</p>
<p>Note that <code>Variable</code>s are deprecated since PyTorch <code>0.4</code> so you can use tensors in newer versions.<br>
Also, call <code>nn.Module</code>s directly via <code>criterion(feature, ...)</code> instead of the <code>.forward</code> method, as the latter approach won’t call into registered hooks and might yield unexpected behavior.</p>

ddguoll: <p>Thank you very much!</p>

ddguoll: <p>l= criterion(feature,target_var, Fea).forward()<br>
l.backward(retain_graph=True)<br>
may be the solution.</p>

ddguoll: <p>I want use the class feature centers in the loss function. But when I compute the centers,<br>
the GPU memory is not enough. How can I solve it? The code is as follows. Thank you very much!</p>
<pre><code class="lang-auto">for epoch in range((args.start_epoch+1), args.epochs):
  Center= computer_Center(model,dataloader, classnum)
	for input, target in train_loader:
        target = target.cuda()
        input = input.cuda()
        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)
        outputs, feature = model(input_var)
        l = criterion(feature,target_var, Center) .forward()
        l.backward(retain_graph=True)

def computer_Center(model,dataloader, classnum):
    model.train()
    for i in range(classnum):
            j=0
            for input,target in dataloader:
                target=target.cuda()
                input = input.cuda()
                input_var = torch.autograd.Variable(input)
                target=torch.autograd.Variable(target)
                _, feature_ext = model(input_var)
                ind=torch.where(target==i)[0]
                if ind.shape[0]&gt;0:
                    if j==0:
                        feature_mid = feature_ext[ind, :]
                        feature_sum_mid=feature_mid.sum(0)
                    else:
                        feature_mid = feature_ext[ind, :]
                        feature_sum_mid = feature_sum_mid+feature_mid.sum(0)
                    j=j+1

            feature_sum_mid=feature_sum_mid.unsqueeze(0)
            if i==0:
                feature_sum=feature_sum_mid
            else:
                feature_sum=torch.cat([feature_sum,feature_sum_mid],dim=0)

    Center=feature_sum
    for i in range(7):
            Center[i,:]=feature_sum[i,:]/ClaSamNum[i]

return Center

</code></pre>

ptrblck: <p>In you code snippet you are accumulating the model output <code>feature_ext</code> in <code>feature_sum_mid</code>, which will also store the computation graph (including all intermediate tensors).<br>
If you want to use this <code>Center</code> tensor as a constant target, you should wrap the calculation in <code>compute_Center</code> into <code>with torch.no_grad()</code> to avoid storing the computation graphs and thus lower the memory usage.</p>
<p>Also, <code>Variable</code>s are deprecated since PyTorch <code>0.4</code>, so you can use tensors in newer versions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ddguoll: <p>Thank you very much. I want to compute the loss for the Center. The Center is not a constant target. Now I computer the Center in the batch-wise. But the performance is not good.</p>

ptrblck: <p>Even though you are calculating the center manually, does Autograd need to backpropagate through these operations?<br>
Compare it to the target of a classification use case. While the target tensor is of course not a constant value, it’s a constant in the sense that Autograd will use it to calculate the loss, and based on this compute the gradient for the parameters in the model, which created the prediction.</p>
<p>Would the same use case be applied here?</p>

ddguoll: <p>I see. Thank you very very very much!</p>