Replace diagonal elements with vector

gelazari: <p>I have been searching everywhere for something equivalent to the following in PyTorch, but I cannot find anything.</p>
<blockquote>
<p>L_1 = np.tril(np.random.normal(scale=1., size=(D, D)), k=0)</p>
</blockquote>
<blockquote>
<p>L_1[np.diag_indices_from(L_1)] = np.exp(np.diagonal(L_1))</p>
</blockquote>
<p>I guess there is no way to replace the diagonal elements in such an elegant way using Pytorch.</p>

SimonW: <pre><code class="lang-auto"># A has size k * k
k = A.size(0)
A.as_strided([k], [k + 1]).copy_(vector)
</code></pre>

gelazari: <p>Thanks <a class="mention" href="/u/simonw">@SimonW</a>. I guess it works in a similar fashion in case of Variables, i.e. my_variable.data()…</p>

SimonW: <p>Please just directly operate on Variable rather than <code>var.data</code> if you want to trace history (and do backward etc).</p>

gelazari: <aside class="quote no-group" data-username="SimonW" data-post="2" data-topic="15311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>k = A.size(0)<br>
A.as_strided([k], [k + 1]).copy_(vector)</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/simonw">@SimonW</a> I’m getting the error “AttributeError: ‘Variable’ object has no attribute ‘as_strided’” although I have the latest PyTorch version. What is the problem?</p>

SimonW: <p>oh i see… Yeah, it might not be available in 0.3.1. Before the next release, you can try advanced indexing <code>A[[1,2,3], [1,2,3]]=4</code> but that might break backward if some graph depends on the original overwritten values. Or you can multiply by a matrix then add another matrix… Yeah, these are not ideal…</p>

gelazari: <p>So, is there any efficient way to do the replacement in PyTorch?</p>

albanD: <p>Hi,</p>
<p>The solution with advances indexing is the way to go for 0.3.1 I think.<br>
Keep in mind that inplace operations are not always possible when working with <code>Variable</code>s because the original value might be needed to compute the backward pass.</p>

gelazari: <p>The thing is that my torch version is ‘0.3.1.post2’ and I do not seem to have the above mentioned functions.</p>

albanD: <p>You should be able to do: <code>A[[1,2,3], [1,2,3]]=4</code> or <code>A[range(size), range(size)] = your_diag_value</code>.</p>

gelazari: <p>Will it work if the “diagonal_value” is a vector though?</p>

albanD: <p>Yes it will work if <code>your_diag_value</code> is a 1D tensor.</p>

gelazari: <p>ok thank you. This will not affect the backward pass right?</p>

albanD: <p>It will affect it because the diagonal values of the original matrix are not used to compute the output anymore.<br>
See the code sample below:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

size = 10
full = Variable(torch.rand(size, size), requires_grad=True)
new_diag = Variable(torch.rand(size), requires_grad=True)

# Do this because we cannot change a leaf variable inplace
full_clone = full.clone()

full_clone[range(size), range(size)] = new_diag

full_clone.sum().backward()

# Should be a diagonal of 0 and 1 for the rest
print(full.grad)
# Should be full of 1
print(new_diag.grad)
</code></pre>

gelazari: <p>So, there is no way to change the diagonal values and keep the backward pass unaffected? Seems strange.</p>

albanD: <p>Why do you want to do that? The gradients that you compute will be completely wrong then…</p>

gelazari: <p>I want to force the diagonal of my covariance matrix to be positive and differentiable wrt the backward pass.</p>

albanD: <p>Ok, not sure that makes sense…<br>
But here is the code to do it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

size = 10
full = Variable(torch.rand(size, size), requires_grad=True)
new_diag = Variable(torch.rand(size), requires_grad=True)

# Do this because we cannot change a leaf variable inplace
full_clone = full.clone()

# WARNING: using data here will break the graph and this
# operation will not be tracked by the autograd engine.
# Hence giving "wrong" gradients
full_clone.data[range(size), range(size)] = new_diag.data

full_clone.sum().backward()

# Should be full of 1
print(full.grad)
# Should be None (equivalent to full of 0)
print(new_diag.grad)
</code></pre>

gelazari: <p>I was doing something like this:</p>
<blockquote>
<pre><code>    self.L_1 = Parameter(torch.randn(dim, dim), requires_grad=True)
    self.L_1.data = torch.tril(self.L_1.data)
    self.log_diag = Parameter(torch.diag(self.L_1.data), requires_grad=True)
    self.log_diag.data = torch.exp(self.log_diag.data)
    self.mask = Parameter(torch.diag(torch.ones_like(self.log_diag.data)))
    self.L = Parameter(self.mask.data * torch.diag(self.log_diag.data) + (1. - self.mask.data) * self.L_1.data, requires_grad=True).cuda()
</code></pre>
</blockquote>

SimonW: <p>If the backward doesn’t need content of that cov matrix you have, then just modifying inplace is fine. (Run .backward to find out.) Otherwise, you can do a clone and then modify inplace.</p>