Train only part of variable in the network

dem123456789: <p>Is there a way to train only part of a variable? I know we can set require_grad = False to a variable. But I want to train only part of the variable. Is there a way to do this?</p>

SimonW: <p>Separate your parameter into to tensors and cat then together.</p>

dem123456789: <p>using torch.split()?</p>

dem123456789: <p>For example</p>
<pre><code class="lang-auto">param = list(model.parameters())
optimizer = torch.optim.SGD(param[0][:10,], lr = 0.01, momentum=0.9)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD([list(torch.split(param[0],10,0))[0], lr = 0.01, momentum=0.9)
</code></pre>
<p>It returns ValueError: can’t optimize a non-leaf Variable<br>
I find out that after I split the parameters, it become a non-leaf node<br>
What I want is to dynamically train sub newtork. Like [<a href="https://discuss.pytorch.org/t/requires-grad-true-false-dynamically/7517?u=dem123456789">requires_grad=True/False dynamically</a>]<br>
I want to split the parameter into even smaller parts. Is there a way to do that?</p>

SimonW: <p>I meant that keeping them as separate tensors, and before activation concatenating them together to use as a single parameter.</p>

dem123456789: <p>I understand what you are describing. I have to declare separated leaf-node parameters and concat them together and them put them into a module. Thus, we ll have leaf-node separated parameters.<br>
So it is not possible to modify nn.module leaf-node parameters?<br>
Things like nn.Llinear, nn.Conv2D once created cannot be separated and trained later?</p>

SimonW: <p>I’m not sure what you mean by separating parameters, but you can achieve it using <code>regsiter_forward_pre_hook</code>, or just use the <code>torch.nn.functonal.*</code> functional forms.</p>

dem123456789: <p>For example<br>
<code>model = nn.Linear()</code><br>
model.parameters() return a tuple of weight and bias. If I want to split the weight into several chunks along dimension 0. I can use<br>
<code>list(model.parameters())[0][:chunk_size,]</code><br>
or<br>
<code>list(torch.split(param[0],chunk_size,0))[0]</code><br>
But both of them will return a non-leaf node which cannot be the input of an optimizer.<br>
One way to resolve is to write my own Linear() module and make weight isolated in advance. Then cat them together into <code>self.weight</code> and finally do things like</p>
<pre><code class="lang-auto">F.linear(input, self.weight, self.bias)
</code></pre>
<p>My question is whether I can just do some magic to the parameters of nn.Linear() that pytorch provided and achieve this goal. The reason is that it is tedious to write every module (nn.Con2d etc.) once again.</p>

SimonW: <p>No, do not split existing parameters. Define your own ones and cat them together in forward or hooks.</p>

SimonW: <p>this might be a good example <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/weight_norm.py#L21">https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/weight_norm.py#L21</a></p>

dem123456789: <p>Thanks for ur inference.<br>
I ll dig into it.<br>
Another potential problem is that I need set all splitted parameters requires_grad = True. But I do not want to compute grad for some of the parameters occasionally.<br>
For example,</p>
<pre><code class="lang-auto">param = [param0 param1]
optimizer = torch.optim.SGD(param0, lr = 0.01, momentum=0.9)
</code></pre>
<p>I dont want to compute grad for param1. But both of them have requires_grad = True<br>
If I do</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>This is related to [<a href="https://discuss.pytorch.org/t/requires-grad-true-false-dynamically/7517/2?u=dem123456789">requires_grad=True/False dynamically</a>]<br>
However, in my case both param0 and param1 are involed in computing the loss function. Will the grad of param1 also be computed? How can I avoid that?<br>
I realize there is a detach_() may be useful. But there is not an attach() method. [<a href="https://github.com/pytorch/pytorch/pull/6561" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/6561</a>]</p>

SimonW: <p>You can either change the flag before forward or just use <code>torch.autograd.grad</code> to specify the inputs for which grad should be computed</p>

dem123456789: <p>Yeah.<br>
Thats one way to do it. With <code>only_intputs = True</code>, it is possible to compute param0 only.<br>
Should I also set <code>retain_graph = True</code> for the optimizer to step?<br>
Is there any difference between using <code>.grad()</code> and <code>.backward()</code> except that  <code>.grad()</code> returns the gradients?</p>

JuliousHurtado: <p>I don’t know if there any “not wanted secondary effect” of this, but maybe you can multiply the grad variable by zero in those part of the variable that you don’t want to modify. Just after the backward part, and before the optimizer step</p>

dem123456789: <p>I wonder whether there is a significant computation cost for those parameters I am not interested.<br>
I have not got a time to dig into automatic differentiation.<br>
For now torch.autograd.grad is not working in this situation.<br>
If I set <code>only_inputs = True</code> (default), the gradient is not accumulated for all parameters and loss does not decrease.<br>
If I set <code>only_inputs = False</code>, the gradient of parameters that I do not put into optimizer are also computed and accumulated because they are all leaf-nodes and have <code>requires_grad = True</code>.<br>
For now I assume that I have to wait pytorch update attach()(or other similar method) to enable dynmaically optimizing selected features.<br>
[<a href="https://github.com/pytorch/pytorch/pull/6561" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/6561</a>]</p>

dem123456789: <p>From pytorch 4.0.<br>
Thanks to <code>requires_grad_()</code> .</p>

Leaves: <p>Hi, <a class="mention" href="/u/dem123456789">@dem123456789</a>, Do you solve the problem of setting part of weights in a layer to require_grad=False?</p>

dem123456789: <p>No. As soon as you concatenate or stack two parameter tensors into one, the gradient of them will be calcuated together.</p>