Can I put the input tensor back to computation graph?

xcwanAndy: <p>Hi, I have a question related to autograd. After finishing FP, I want to transfer the input tensor from GPU to CPU, and then after a while transfer it back to GPU for backward and obtain grads. The problem is that if I transfer the tensor back, it won’t go back to the computation graph, and therefore unable to obtain grads from the tensor.</p>
<p>I know there is way that I hook the original GPU tensor to obtain grad. But I wonder if it’s possible to obtain the grad in the transferring back stage.</p>
<p>Here is a simple example:</p>
<pre><code class="lang-auto">import torch

device = torch.device("cuda:0")
cpu = torch.device("cpu")

a = torch.randn(5, requires_grad=True, device=device)
b = torch.randn(5, requires_grad=True, device=device)

with torch.autograd.graph.save_on_cpu():
    d = a * b

grads = []

def hook_wrapper(grads):
    def hook(input_gradient):
        grads.append(input_gradient)

    return hook

a = a.to(cpu)
a = a.to(device)
a.register_hook(hook_wrapper(grads))

d.sum().backward()

print(len(grads))
print(grads)
</code></pre>
<p>where the len(grads) will be 0 and I cannot obtain the grad.</p>

xcwanAndy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Would you please give me some suggestions? Thanks!</p>

J_Johnson: <p>A better way I’ve found to change the graph after the fact is to:</p>
<ol>
<li>.detach() the model outputs,</li>
<li>use those to alter the targets(these aren’t in the graph and can be altered at this stage before the loss function),</li>
<li>Run your loss function on the altered targets.</li>
</ol>
<p>You won’t need to take the outputs or targets off GPU. Just keep all of the tensors on GPU and run your calcs there for speed.</p>

xcwanAndy: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> Thanks for your reply. But the fact in my case is that the output tensors are too large to be kept in GPU. And I have to trade some outputs out of GPUs to preserve space for other batches’ computation. BTW, my case is that I should run all batches forward, and then run them backward, so GPU memory is key constraint.</p>
<p>Do you have any suggestions for this case?</p>

J_Johnson: <p>The graph ends up taking 2/3s or more of the total memory during training. Some optimizers more and some less. I’d be surprised to see the outputs larger than the model and the gradients. Are you storing all of the outputs during training?</p>
<p>Do you have a second GPU? You may want to clock the time it takes for each part in the process to ensure you are fine with any latency overhead.</p>
<p>Without knowing more about what you’re trying to accomplish, that is all I have for suggestions.</p>

xcwanAndy: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a>  Thanks. Let me explain my process more specifically. I want to achieve this process:</p>
<p>For a given large dataset, I slice the dataset into minibatches, and run them batch by batch in the forward process. (Yes, no batch of data will run backward before all batches finish, which result in a lot of computation graphs reside in GPUs, and that’s why I use <code>torch.autograd.graph.save_on_cpu</code> to put the generated tensors back to CPU). After all batches finish FP, I will place the output tensors from CPU back to GPU and run BP.</p>
<p>My current question is that: In the backward pass, I don’t know how to put the output tensor from CPU back to the corresponding computation graph place in GPU. I think PyTorch might preserve certain hook APIs for me to achieve this, and I wonder if you could give me any suggestion.</p>

J_Johnson: <p>With Stochastic Gradient Descent(SGD) you’re just subracting the gradient from each weight/bias on the fly.</p>
<aside class="onebox wikipedia" data-onebox-src="https://en.m.wikipedia.org/wiki/Stochastic_gradient_descent">
  <header class="source">

      <a href="https://en.m.wikipedia.org/wiki/Stochastic_gradient_descent" target="_blank" rel="noopener nofollow ugc">en.m.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:220/173;"><img src="//upload.wikimedia.org/wikipedia/commons/thumb/f/f3/Stogra.png/220px-Stogra.png" class="thumbnail" width="220" height="173"></div>

<h3><a href="https://en.m.wikipedia.org/wiki/Stochastic_gradient_descent" target="_blank" rel="noopener nofollow ugc">Stochastic gradient descent</a></h3>

<p>Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Why not just get the gradients for each batch and sum those together(so you just have one set of gradients) and then manually subtract those(times a learning rate, I’d set this much higher, since you’re running it all at once) from your parameters once your data has all been run through the model? That is what you’re effectively doing, anyway. I mean, you have to keep each layer of gradients separate. Don’t sum across layers, just across batches.</p>
<p>You won’t need momentum, since you are doing it all in one go.</p>
<p>Because you’re adding them at each iteration, the overhead would be equal to your model size. So you could keep it on GPU.</p>

xcwanAndy: <p>Yes, right… But for some reason (e.g., samples have dependencies between each other), I have to finish whole dataset’s FP first and then backward. Would there be any suggestions for me to trade the output tensors from GPU and then back to GPU afterwards?</p>

J_Johnson: <p>If the data inputs are sequentially dependent on each other(such as in a time series), perhaps what you need is a recurrent neural network(i.e. LSTM, GRU, etc.) or a transformer. RNNs have a gated memory in the layer, good for the model to learn sequential information. Transformers, on the other hand, keep positional information of the sequence, and allow for more asynchronuous processing of sequential information. I.e. you might put in the last frame as inputs in an RNN, while in a Transformer, you’d put in 60 frames as one input(batches would then be windows of 60 frames while just randomizing which window is used).</p>
<p>What you’re asking might not be possible as autograd stores the operations in a DAG during forward propagation. You can read more about autograd here:</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#computational-graph" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#computational-graph</a></p>
<p>But I think if you become more familiar with the math behind the scenes, you’ll find there are more elegant solutions to your problem.</p>