Variable update after autograd in torch.cat, torch.vstack, nn.ZeroPad2d, torch.index_select and torch.roll

username: Amine08
cooked: <p>Hi,</p>
<p>let x=<strong>torch.vstack</strong>((a,b)) and a is being optimized using autograd. How can we update the new value of x without using x=torch.vstack((a,b)) again, i.e. the new value of a will be updated automatically in x? (using shared memory or something like this)</p>
<p>Same question for <strong>nn.ZeroPad2d,</strong> <strong>torch.index_select,</strong> <strong>torch.cat,</strong>  <strong>torch.roll</strong> and <strong>in-place operation</strong></p>
<p>PS: e.g. torch.flatten updates the values properly</p>
<p>Here an example:</p>
<blockquote>
<p>import torch<br>
import torch.optim as optim</p>
<p>x = torch.randn(3, 4,requires_grad=True)<br>
y=torch.cat((x,x))<br>
L = torch.randn(3, 4)<br>
L[0,0]=x[0,0]<br>
f=torch.flatten(x)</p>
<p>t=y.sum()<br>
t.backward()<br>
opt = optim.Adam(<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>, lr=0.1)<br>
print(“x”,x)<br>
print(“f”,f)<br>
print(“y”,y)<br>
print(“L”,L)<br>
opt.step()<br>
print(“========================”)<br>
print(“========================”)<br>
print(“x”,x)<br>
print(“f”,f)<br>
print(“y”,y)<br>
print(“L”,L)</p>
</blockquote>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: soulitzer
cooked: <p>If you begin with two separate tensors they will live in different places in memory so it is not possible to combine them into a single tensor without copying them both into new tensor.</p>
<p>One trick to do this could be to first begin with the big tensor, and then take small views of that tensor.</p>
<pre><code class="lang-auto">import torch
import torch.optim as optim

# some dummy tensors that have the same shape as `a` used to initialize `x`
_a = torch.tensor(1.)
_b = torch.tensor(2.)
x = torch.vstack((_a, _b))

# now we finally create `a`
a = x.select(0, 0)
a.requires_grad_(True) # a.is_leaf is True
opt = optim.Adam([a]) # optimize `a`

loss = a ** 2
loss.backward()

print(a)
print(x)

opt.step()

print(a)
print(x) # should be updated as well
</code></pre>
<p>The trick is basically to find the inverse operation of whatever you’re doing. So for arbitrary <code>fn</code> (where <code>x = fn(a)</code>), make sure you find <code>fn2</code> such that `a = fn2(fn(a))</p>

username: Amine08
cooked: <p>Hi,</p>
<p>the problem is that instead of _a and _b i have the model parameters which are already defined and they should also be optimized (in this case _a und _b do not, see my small modification to you code). So i want as a summary that the model weights are getting optimized using autograd and my matrix contains the new weight values automatically.</p>
<p>This code with the small modification works but with a convolution layer it is complicated or even not possible, because my matrix contains the 2D representation of the weight tensor plus some modifications (padding between the rows and columns etc), e.g. let w0,w1,w2,w3 be the model weights (convolutional layers with many channels) than my matrix is e.g. defined as [[W0,W1],[W2,W3]] s.t. W0 is a modification of w0 incl. reshaping to get 2D matrix + padding between the rows and columns etc, and my goal ist that W0,W1,W2,W3 get updated when w0,w1,w2,w3 do. I don’t want to define every time my matrix to save execution time.</p>
<p>your modified code:</p>
<pre><code class="lang-auto">import torch
import torch.optim as optim
# some dummy tensors that have the same shape as `a` used to initialize `x`
_a = torch.tensor(1.)
_b = torch.tensor(2.)
x = torch.vstack((_a, _b))

# now we finally create `a`
a = x.select(0, 0)
_a=a
a.requires_grad_(True) # a.is_leaf is True
opt = optim.Adam([a]) # optimize `a`

loss = a ** 2
loss.backward()

print(_a)
print(a)
print(x)

opt.step()

print(_a) # should be updated as well
print(a)
print(x) # should be updated as well
</code></pre>
<p>Here an example of my goal:</p>
<pre><code class="lang-auto">import torch
import torch.optim as optim

# original weight matrix
a = torch.tensor([[[1.,2],[3.,4]], 
                  [[1.,1],[2.,2]]]) # shape [2,2,2]


# create a modificated weight matrix from the original one
a_new=torch.cat((a[0],torch.zeros(2,2),a[1])) #shape[6,2]



a.requires_grad_(True) # a.is_leaf is True
opt = optim.Adam([a]) # optimize `a`

loss = a.sum()
loss.backward()

print("a",a)
print("a_new",a_new)

opt.step()
print("================")
print("================")


print("a",a)
print("a_new",a_new) # should be updated as well
</code></pre>