Require gradient only for some tensor elements (others should be held constant)

Dominik: <p>I like to use a tensor with only a few variable elements which are considered during the backpropagation step. Consider for example:</p>
<pre><code class="lang-auto">a = torch.ones((2, 2), requires_grad=True)
b = torch.ones((2, 2), requires_grad=True)
c = torch.ones((1, 2), requires_grad=True)

x = torch.ones((2, 1))

y = c @ (b @ (a @ x))
</code></pre>
<p>Now for the <code>c</code> tensor I want the <code>c[0, 1]</code> element to be constant, i.e. not being varied during the optimization procedure. One idea is that, after doing e.g. <code>y.sum().backward()</code>, I could zero the corresponding gradient elements:</p>
<pre><code class="lang-auto">c.grad[0, 1] = 0
</code></pre>
<p>Is this a correct way of dealing with the problem? Or are the other gradients (of <code>a, b</code>) being influenced by the value of <code>c.grad[0, 1]</code> during the backpropagation step? Considering the gradient calculation it seems they shouldn’t:</p>
<p><img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20w_%7Bjk%7D%5Ei%7D%20%3D%20%5Cfrac%7BL%7D%7B%5Cpartial%20a_j%5Ei%7D%5Ccdot%5Cfrac%7B%5Cpartial%20a_j%5Ei%7D%7B%5Cpartial%20w_%7Bjk%7D%5Ei%7D%20%5Cstackrel%7B%21%7D%7B%3D%7D%200" alt width="157" height="48"></p>
<p>where <code>L = y.sum()</code>, <code>a^i_j</code> is the value of the j-th element in the <code>i-th</code> layer of the computational graph (e.g. <code>a^1_0 == (a @ x)[0]</code>) and <code>w^i_{jk}</code> is the <code>jk</code> element of the i-th tensor (e.g. <code>w^2_{01} == b[0, 1]</code>. The first term of the above product is passed on during backpropagation and since it doesn’t contain derivatives with respect to any weights it seems that modifying parts of the gradient of a specific tensor wouldn’t affect previous gradients in the graph.</p>
<p>I just want to double check on this and ask in general if this is the preferred way of dealing with such situations? Maybe there is a more elegant (more appropriate) way of dealing with constant tensor elements?</p>
<p>Any help is appreciated. Thanks.</p>

pranavkrai: <p>Hi,<br>
Were you able to figure out a solution of this topic? I’ve been struggling with this issue on tensorflow and using some ad-hoc methods to deal with them.</p>

Dominik: <p>I ended up using a scalar tensor with <code>requires_grad=True</code> and then adding it to the overall constant tensor element like this:</p>
<pre><code class="lang-python">x = torch.tensor(1., requires_grad=True)
y = torch.zeros((2, 2))
y[0, 1] += x
</code></pre>
<p>Now this causes all elements of <code>y</code> to be constant except for the <code>(0, 1)</code> element. If you want it the other way around, i.e. all but one element require grad, then you just need more tensors to add. For example let’s say the <code>(0, 1)</code> should be constant, so we can do:</p>
<pre><code class="lang-python">x = torch.ones((2, 1), requires_grad=True)
y = torch.tensor(1., requires_grad=True)
z = torch.zeros((2, 2))
z[:, 0] += x
z[1, 1] += y
</code></pre>
<p>For a 2D tensor (matrix) at most four distinct tensors with <code>requires_grad=True</code> should suffice to make all but a single element require grad as well.</p>
<p>Nevertheless I think that the method of setting the <code>.grad</code> attribute partially to zero works as well.</p>

pranavkrai: <p>Thanks for this answer. Thanks a lot. Could I bother you with one more? Given that I’m going to use the opt.minimize method to minimize a loss function, in tensorflow, one needs to set a var_list argument, which tells the optimizer(SGD etc) which tensors to vary. In that case, i would just add x(and y) to that list, and leave z out, or can i just list z and it can figure out which elements are variables and which are constants ?</p>

Dominik: <p>I can only respond from the PyTorch perspective, but here you would make the original tensors (the ones with <code>requires_grad=True</code>) to be the parameters of the optimization. In the end, operations like <code>y[0, 1] += x</code> create a new node in the computation graph, with inputs <code>x</code> and <code>y</code>, where <code>x</code> is variable and <code>y</code> is constant. Only <em>after</em> that node, the resulting tensor <code>y</code> requires grad as well, but there is no such notion for the autograd engine, especially when it comes to the fact that only one element of that tensor should be variable. I suspect the situation is the same for tensorflow.</p>

enterthevoidf22: <p>hey - i think i have a similar situation and wanted to check my method of solving it -<br>
say i have a matrix of weights in a neural network and i want only specific elements in the matrix to be trained using backprop, say the bottom row. so i would initialize the entire matrix as random with requires_grad=True and then multiply by a matrix of zeros with the bottom row set to ones with requires_grad=False.</p>
<p>will this work?</p>

ccoding: <p>The easiest way is adding a mask constant.<br>
Like</p>
<pre><code class="lang-python">y = c@(b@(a@x))
</code></pre>
<p>You can add a constant like <code>mask = torch.ones((2,2))</code><br>
ans set <code>mask[0,1]=0</code><br>
and you can switch your formula to</p>
<pre><code class="lang-python">y = mask.dot(c) @(b@(a@x))
</code></pre>
<p>so the gradient of y will not back propagate to c[0,1]</p>

enterthevoidf22: <p>i didn’t understand what you did with the @ but that’s what i said no?</p>
<p>i want to learn a representation only on top row of a matrix, called embed:</p>
<p>embed = torch.randn((1,10,10),requires_grad=True)<br>
mask =torch.zeros((1,10,10),requires_grad=False)<br>
mask[0,0,:] = 1<br>
embed = embed * mask</p>
<ul>
<li>now build the network with embed where i want it to be.</li>
</ul>
<p>p.s<br>
the reason i want it still to be a matrix is because of the architecture that can only accept a matrix at that point.</p>

ccoding: <p>Yes , you got what I meant.<br>
That’s the easiest way to erase the gradient of your matrix.<br>
I don’t know the ‘@’ mean, I thought  it might be any operator like \times \add \minus in your formula.</p>

Dominik: <p>You can do that, but obviously it will change the results of the computation, if you set some of the elements of <code>embed</code> to zero already in the forward step. It depends on your use case, whether you want the masked elements to contribute to the forward pass or not. Using <code>embed = embed * mask</code> it will set the contributions to zero.</p>
<p>My original use case was different, I had non-zero matrix elements which should contribute to the final result, but not change during training. Hence I used a different approach.</p>

enterthevoidf22: <p>hey, i’m back with this issue of setting only certain elements of the tensor “learnable” -<br>
my net is more complicated than this example but maybe it’s enough to find my issue without describing the whole thing:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(2, 6, 3) #notice two channel input
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(16 * 6 * 6, 2)
        self.embed = torch.randn((1,10,10),requires_grad=True) # lets say the input is a 10x10 image, and so is the embedding before masking
        mask = torch.zeros((1,10,10),requires_grad=False)
        mask[0,:,5] = 1 # we want to mask the embedding such that only one row is actually updated during backprop
        self.embed = self.embed * mask # notice that embed requires_grad and mask doesn't require. is this how it should be done or vice versa?

    def forward(self, x):
        x = torch.cat([self.embed,x])
        out = self.conv1(x)
        out = flatten(out)
        return self.fc1(out)
</code></pre>
<p>should this work in theory?</p>
<p>in my case it’s slightly more complicated, because certain inputs belong to one category and i want to learn an embedding of each category, so the forward pass accepts x and category,it looks like this:</p>
<pre><code class="lang-auto">self.embed =  self.embed.repeat(category_num,1,1)
def forward(self,x, category):
     x = torch.cat([self.embed[category], x])
     out = self.conv1(x)
     out = flatten(out)
     return self.fc1(out)
</code></pre>
<p>i’m doing almost exactly that and getting a weird error that i couldn’t solve -</p>
<pre><code class="lang-auto">runtimeerror: trying to backward through the graph a second time, but the saved intermediate results have already been freed. specify retain_graph=true when calling backward the first time.
</code></pre>

Dominik: <p>The reason you get this error is because you are using the <code>self.embed = self.embed * mask</code> version in <code>forward</code> (I will call this <code>self.embed2</code> in the following, as if you did <code>self.embed2 = self.embed * mask</code>). This tensor object is however not a leaf in the computation graph; the leaf is the original <code>self.embed = torch.randn((1,10,10),requires_grad=True)</code> object (I will continue to call this <code>self.embed</code> in the following). So your computation graph goes like this: <code>self.embed --(a)--&gt; self.embed2 --(b)--&gt; ...</code>. In <code>forward</code> you are effectively using the <code>self.embed2</code> tensor, so the edge <code>--(a)--&gt;</code> never gets renewed, and hence it will remain in the graph also for the second forward pass and hence the error that you attempt to backprop another time through (that part of) the graph. Also note that if this worked (you could make it work by using <code>backward(retain_graph=True)</code>), your updates to the original <code>self.embed</code> tensor would never be effective, since you’re stuck with the <code>self.embed2</code> version (which contains the original values of <code>self.embed</code>).</p>
<p>So the solution is to recompute <code>self.embed2</code> on every <code>forward</code> pass. Basically you need to shift the relevant lines to <code>forward</code>:</p>
<pre><code class="lang-auto">def __init__(self):
    ...
    self.embed = torch.randn((1,10,10),requires_grad=True)
    # Move the `mask`ing part to `forward`.

def forward(self):
    mask = torch.zeros((1,10,10),requires_grad=False)
    mask[0,:,5] = 1
    embed = self.embed * mask  # don't overwrite `self.embed`
    x = torch.cat([embed,x])  # use the newly created `embed` tensor
    ...
</code></pre>

enterthevoidf22: <p>That’s a great explanation, thanks!</p>
<p>What should I do in case my masking tensor is actually a very large tensor with a binary pattern that I load in the argument section of the init function?</p>
<p>Can I somehow project a tensor I load when initiating the neural net onto forward pass?</p>

Dominik: <p>Sure, actually you can instantiate the <code>mask</code> already in <code>__init__</code>, i.e.:</p>
<pre><code class="lang-auto">def __init__(self):
    ...
    self.embed = ...
    self.mask = ...
    self.mask[0,:,5] = 1
    # Don't compute the masked version of 'embed' yet.

def forward(self, x):
    embed = self.embed * mask  # don't overwrite `self.embed`
    x = torch.cat([embed,x])  # use the newly created `embed` tensor
</code></pre>
<p>However any operation that will create a new node in the computation graph (i.e. any operation that involves a tensor with <code>requires_grad=True</code>) needs to happen in forward, so that the changes to the parameters can become effective.</p>

enterthevoidf22: <p>great!<br>
another question, hope it’s alright  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
i notice my gpu utilization is declining rapidly thanks to this embedding procedure, as it causes some bottleneck along the way. can you think of any method of reducing it?</p>
<p>my actual input data is of size (batch=4,channel=1,width=75,height=95,depth=65) -<br>
this means my self.embed is of size (category_num=155,channel=1,width=75,height=95,depth=65)<br>
and so is self.mask.</p>
<p>the init func is now as so:</p>
<pre><code class="lang-auto">        if args.subject_embedding:
            self.do_embed = True
            self.embed = torch.randn((self.n_subjects, 1, self.dim[0], self.dim[1], self.dim[2]), requires_grad=True)
            self.mask = torch.ones(self.embed.shape,requires_grad=False)
            if args.embedding_mask is not None:
                self.mask = args.embedding_mask
                self.mask.requires_grad = False
                if self.mask.shape[0] == 1:
                    self.mask = self.mask.unsqueeze(0).repeat(self.n_subjects, 1, 1, 1, 1)
                else:
                    self.mask = self.mask.unsqueeze(0).unsqueeze(0).repeat(self.n_subjects, 1, 1, 1, 1)
            if args.cuda:
                self.mask = self.mask.cuda()
                self.embed = self.embed.cuda()
</code></pre>
<p>the forward pass is implemented as so:</p>
<pre><code class="lang-auto">    def forward(self, x, subj):

        #  Level 1 context pathway
        #print('input shape ',x.shape)
        context_0 = x
        if self.do_embed:
            embed = self.embed * self.mask
            E = embed[subj.to(dtype=torch.long)]
            x = torch.cat([x,E],dim=1)
</code></pre>
<p>still, since the only nonzero elements of self.mask are 40 per category, i thought this would reduce the bottleneck. but now i see it doesn’t and training will be slow.<br>
notice that i push both the entire huge embedding to cuda and the huge ones tensor to cuda, where effectively i’m not sure that is necessary.<br>
i was wondering if perhaps keeping them in cpu and only pushing to cuda the result of their multiplication (the tensor E ) in the forward pass, would reduce the bottleneck. i implemented it and couldn’t prove myself it’s more efficient yet.</p>
<p>P.S<br>
do not be confused by the initiation of self.mask as a ones tensor, that is for the case the masking feature is turned off and then it operates as an identity transformation.</p>

MonKeyBoy: <p>Hi, I try some codes:</p>
<pre><code class="lang-auto">x = torch.tensor(1., requires_grad=True)
y = torch.zeros((2, 2))
y[0, 1] += x
print(y[0, 1].requires_grad)
print(y[1, 1].requires_grad)
</code></pre>
<p>It should print <strong>True and False</strong>. However, it outputs both <strong>True</strong>.<br>
I am very confused about it. Hope you can help me to explain it.</p>