Changing view of input breaks graph

username: VRehnberg
cooked: <p>I want to compute the gradient between an input and an output. However, when I change the view of the input I get an error message</p>
<blockquote>
<p>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
</blockquote>
<p>What I’ve tried is</p>
<pre><code class="lang-auto">import torch

func = torch.nn.Linear(5, 6)
x = torch.rand(3, 5, requires_grad=True)
for x_row in x.unbind(0):
    y_row = func(x_row)
    torch.autograd.grad(y_row, [x_row], grad_outputs=torch.ones_like(y_row), create_graph=True)[0]
    # This is fine

y = func(x)
for x_row, y_row in zip(torch.split(x, 1, dim=0), torch.split(y, 1, dim=0)):
    torch.autograd.grad(y_row, [x_row], grad_outputs=torch.ones_like(y_row), create_graph=True)[0]
    # RuntimeError

y = func(x)
for x_row, y_row in zip(x.unbind(0), y.unbind(0)):
    torch.autograd.grad(y_row, [x_row], grad_outputs=torch.ones_like(y_row), create_graph=True)[0]
    # RuntimeError


x_row = torch.rand(1, 5, requires_grad=True)
x_row = x_row.flatten()
y_row = func(x_row)
torch.autograd.grad(y_row, [x_row], grad_outputs=torch.ones_like(y_row), create_graph=True)[0]
# This is fine

x_row = torch.rand(1, 5, requires_grad=True)
y_row = func(x_row)
x_row = x_row.flatten()
y_row = y_row.flatten()
torch.autograd.grad(y_row, [x_row], grad_outputs=torch.ones_like(y_row), create_graph=True)[0]
# RuntimeError
</code></pre>
<p>How am I supposed to do this?</p>

username: VRehnberg
cooked: <p>So the problem is probably quite obvious. It has to do with the direction of the DAG of gradients.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7bdaa9fd982a551ac61530e936ee15ea5f7c682.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7bdaa9fd982a551ac61530e936ee15ea5f7c682" title="Namnlös teckning"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7bdaa9fd982a551ac61530e936ee15ea5f7c682_2_666x500.png" alt="Namnlös teckning" data-base62-sha1="qdrJC1nvUU6Mvz0V04B2EmBakZI" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7bdaa9fd982a551ac61530e936ee15ea5f7c682_2_666x500.png, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7bdaa9fd982a551ac61530e936ee15ea5f7c682.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7bdaa9fd982a551ac61530e936ee15ea5f7c682.png 2x" data-dominant-color="BDC1BF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Namnlös teckning</span><span class="informations">960×720 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>A possible way to rework this is</p>
<pre><code class="lang-auto">import torch

func = torch.nn.Linear(5, 6)
x = torch.rand(3, 5, requires_grad=True)
x_rows = x.unbind(0)
x = torch.vstack(x_rows)
y_rows = func(x).unbind(0)
for x_row, y_row in zip(x_rows, y_rows):
    torch.autograd.grad(y_row, [x_row], grad_outputs=torch.ones_like(y_row), create_graph=True)[0]
</code></pre>
<p>However, I wouldn’t consider this to be a solution as this depends on having access to what is put into the model. The best outcome would be if the experimental feature <a href="https://pytorch.org/docs/master/generated/torch.vmap.html" rel="noopener nofollow ugc">vmap</a> would turn out to work together with autograd.grad.</p>