How does one have the parameters of a model NOT BE LEAFS?

pinocchio: <p>I want the parameters of the model to NOT be leafs. I also want the parameters to be part of the computation graph and I am dynamically changing the parameters with code. So I want:</p>
<ul>
<li>Parameters to NOT be leafs (but still be held inside a <code>nn.Module</code> so that I can use the dynamically changing parameters as part of the computation graph).</li>
<li>I want the “updating model” to be part of the computation graph at any time but it seems setting the parameters to leaf is stopping the flow of gradients</li>
</ul>
<p>How can I do that?</p>
<p>(plan to post later a working self-contained simplified jupyter demo of what more or less is going to make it easier to help me)</p>
<hr>
<p>SO: <a href="https://stackoverflow.com/questions/60271131/how-does-one-have-parameters-in-a-pytorch-model-not-be-leafs-and-be-in-the-compu" rel="nofollow noopener">https://stackoverflow.com/questions/60271131/how-does-one-have-parameters-in-a-pytorch-model-not-be-leafs-and-be-in-the-compu</a></p>

albanD: <p>Hi,</p>
<p>The general trick to make sure they are not leafs anymore is to first delete de field containing an nn.Parameter. Then you can set it with a Tensor that is not a leaf.</p>

pinocchio: <p>tried what you suggested but it didn’t work…</p>
<pre><code class="lang-auto">        del loss_net.fc0.weight
        #setattr(loss_net.fc0,'weight', wt)
        loss_net.fc0.weight = wt
</code></pre>
<p>what did you have in mind?</p>
<hr>
<p>Error msg:</p>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>

pinocchio: <p>variations that don’t work:</p>
<pre><code class="lang-auto">        #del loss_net.fc0.weight
        #setattr(loss_net.fc0, 'weight', nn.Parameter( wt ))
        #setattr(loss_net.fc0, 'weight', wt)
        #loss_net.fc0.weight = wt
        #loss_net.fc0.weight = nn.Parameter( wt )
</code></pre>

albanD: <p>Hi,</p>
<p>This does work:</p>
<pre><code class="lang-python">m = nn.Linear(10, 10)
del m.weight
m.weight = torch.rand(10)
</code></pre>
<p>If you try to print the weight after the del, you should get that you module has no attribute ‘weight’, is that what you get?</p>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="5" data-topic="70076">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>If you try to print the weight after the del, you should get that you module has no attribute ‘weight’, is that what you get?</p>
</blockquote>
</aside>
<p>yes, I get that error msg:</p>
<pre><code class="lang-auto">AttributeError: 'Linear' object has no attribute 'weight'
</code></pre>

pinocchio: <p>But I get another error now:</p>
<pre><code class="lang-auto">RuntimeError: OrderedDict mutated during iteration
</code></pre>
<p>I am looping over all the parameters of my (potentially arbitrary) model. So, I guess python gets unhappy that I’ve mutated it…</p>

pinocchio: <p><a class="mention" href="/u/alband">@albanD</a></p>
<p>This is the self contained dummy example/prototype I built to debug this issue:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

import copy

from collections import OrderedDict

# img = torch.randn([8,3,32,32])
# targets = torch.LongTensor([1, 2, 0, 6, 2, 9, 4, 9])
# img = torch.randn([1,3,32,32])
# targets = torch.LongTensor([1])
x = torch.randn(1)
target = 12.0*x**2

criterion = nn.CrossEntropyLoss()

#loss_net = nn.Sequential(OrderedDict([('conv0',nn.Conv2d(in_channels=3,out_channels=10,kernel_size=32))]))
loss_net = nn.Sequential(OrderedDict([('fc0', nn.Linear(in_features=1,out_features=1))]))

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('fc0',nn.Linear(in_features=1,out_features=1))]))
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
#
nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^&lt;t&gt; := f(w^&lt;t-1&gt;,delta^&lt;t-1&gt;)
    for (name, w) in loss_net.named_parameters():
        print(f'name = {name}')
        print(w.size())
        hidden = updater_net(hidden).view(1)
        print(hidden.size())
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        print(wt.size())
        new_params[name] = wt
        #del loss_net.fc0.weight
        #print(loss_net.fc0.weight)
        #setattr(loss_net.fc0, 'weight', nn.Parameter( wt ))
        #setattr(loss_net.fc0, 'weight', wt)
        #loss_net.fc0.weight = torch.randn(1)
        #loss_net.fc0.weight = nn.Parameter( wt )
    ##
    loss_net.load_state_dict(new_params)
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}')
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')
</code></pre>
<p>Hope it helps us…</p>
<p>its current (incorrect) output:</p>
<pre><code class="lang-auto">updater_net.fc0.weight.is_leaf = True
i = 0
name = fc0.weight
torch.Size([1, 1])
torch.Size([1])
torch.Size([1, 1])
name = fc0.bias
torch.Size([1])
torch.Size([1])
torch.Size([1])
i = 1
name = fc0.weight
torch.Size([1, 1])
torch.Size([1])
torch.Size([1, 1])
name = fc0.bias
torch.Size([1])
torch.Size([1])
torch.Size([1])

updater_net.fc0.weight.is_leaf = True

-- params that dont matter if they have gradients --
loss_net.grad = tensor([[9.6197]])
-- params we want to have gradients --
hidden.grad = None
updater_net.fc0.weight.grad = None
updater_net.fc0.bias.grad = None
</code></pre>

albanD: <p>Hi,</p>
<p><code>load_state_dict()</code> load the weights in a non-differentiable manner. So you end up with leafs as expected.</p>
<p>Here is a version that works as expected:</p>
<pre><code class="lang-python">
def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^&lt;t&gt; := f(w^&lt;t-1&gt;,delta^&lt;t-1&gt;)
    for (name, w) in list(loss_net.named_parameters()):
        hidden = updater_net(hidden).view(1)
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        del_attr(loss_net, name.split("."))
        set_attr(loss_net, name.split("."), wt)
    ##
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
print(f'loss_net.fc0.weight.is_leaf = {loss_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}') # None because this is not a leaf, it is overriden in the for loop above.
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')
</code></pre>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="9" data-topic="70076">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>def del_attr(obj, names): if len(names) == 1: delattr(obj, names[0]) else: del_attr(getattr(obj, names[0]), names[1:]) def set_attr(obj, names, val): if len(names) == 1: setattr(obj, names[0], val) else: set_attr(getattr(obj, names[0]), names[1:], val)</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/alband">@albanD</a> oh wow! The grads are being printed for sure now! Who also but the albanD boss to come to rescue…</p>
<p>Sorry for being demanding, but is it ok if you comment or explain what you did that made things work? Especially this code:</p>
<pre><code class="lang-auto">def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)
</code></pre>
<p>Thanks so much again…your a saviour… <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=12" title=":muscle:" class="emoji" alt=":muscle:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

pinocchio: <p>self contained script that seems to work:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

from torchviz import make_dot

import copy

from collections import OrderedDict

# img = torch.randn([8,3,32,32])
# targets = torch.LongTensor([1, 2, 0, 6, 2, 9, 4, 9])
# img = torch.randn([1,3,32,32])
# targets = torch.LongTensor([1])
x = torch.randn(1)
target = 12.0*x**2

criterion = nn.CrossEntropyLoss()

#loss_net = nn.Sequential(OrderedDict([('conv0',nn.Conv2d(in_channels=3,out_channels=10,kernel_size=32))]))
loss_net = nn.Sequential(OrderedDict([('fc0', nn.Linear(in_features=1,out_features=1))]))

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('fc0',nn.Linear(in_features=1,out_features=1))]))
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
#
def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^&lt;t&gt; := f(w^&lt;t-1&gt;,delta^&lt;t-1&gt;)
    for (name, w) in list(loss_net.named_parameters()):
        hidden = updater_net(hidden).view(1)
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        del_attr(loss_net, name.split("."))
        set_attr(loss_net, name.split("."), wt)
    ##
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
print(f'loss_net.fc0.weight.is_leaf = {loss_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}') # None because this is not a leaf, it is overriden in the for loop above.
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')
make_dot(loss_val)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">updater_net.fc0.weight.is_leaf = True
i = 0
i = 1

updater_net.fc0.weight.is_leaf = True
loss_net.fc0.weight.is_leaf = False

-- params that dont matter if they have gradients --
loss_net.grad = None
-- params we want to have gradients --
hidden.grad = None
updater_net.fc0.weight.grad = tensor([[0.7152]])
updater_net.fc0.bias.grad = tensor([-7.4249])
</code></pre>

albanD: <p>Since you have arbitrary name, but you need to be able to delete and set attributes (to delete the nn.Parameter and set a Tensor instead). So this is a recursive function that given a list of all the names like [“foo”, “bar”, “weight”] will either set or delete obj.foo.bar.weight.</p>

pinocchio: <p>AlbanD, this does quite work. I wanted to make sure it worked by printing the computation graph and increasing the <code>nb_updates</code> should increase the computation graph but it doesn’t. I believe it has something to do with the fact we are changing/mutating the OrderDict/params <strong>and</strong> looping through it at the same time. I tried to fix it by collecting the parameters before and after running the loop but the somehow it thinks its empty for the second loop (i.e. <code>t=1</code> doesn’t print anything when it should):</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

from torchviz import make_dot

import copy

from collections import OrderedDict

def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
        
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

        
x = torch.randn(1, requires_grad=True)
y = torch.randn(1, requires_grad=True)

criterion = nn.CrossEntropyLoss()

loss_net = nn.Sequential(OrderedDict([('l_fc0', nn.Linear(in_features=1,out_features=1, bias=True))]))
loss_net.l_fc0.weight.requires_grad=False
loss_net.l_fc0.bias.requires_grad=False

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('u_fc0',nn.Linear(in_features=1,out_features=1))]))
updater_net.u_fc0.bias.requires_grad = False
print(f'updater_net.u_fc0.weight.is_leaf = {updater_net.u_fc0.weight.is_leaf}')
#
outputs_virgin = loss_net(x)
params = dict(dict(loss_net.named_parameters()),**{'x':x})
make_dot(outputs_virgin, params=params).render('loss_net_x', format='png')
#
nb_updates = 2
params = list(loss_net.named_parameters())
for t in range(nb_updates):
    print(f't = {t}')
    ## w^&lt;t&gt; := f(w^&lt;t-1&gt;,delta^&lt;t-1&gt;)
    for (name, w) in params:
        delta = updater_net(hidden).view(1)
        wt = w + delta
        print(f'w^&lt;{t}&gt; = {wt}')
        del_attr(loss_net, name.split("."))
        set_attr(loss_net, name.split("."), wt)
    params = list(loss_net.named_parameters())

print()
print(f'updater_net.u_fc0.weight.is_leaf = {updater_net.u_fc0.weight.is_leaf}')
print(f'loss_net.l_fc0.weight.is_leaf = {loss_net.l_fc0.weight.is_leaf}')

outputs = loss_net(x)
loss_val = (outputs - y)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.l_fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}') # None because this is not a leaf, it is overriden in the for loop above.
print(f'updater_net.u_fc0.weight.grad = {updater_net.u_fc0.weight.grad}')
#print(f'updater_net.u_fc0.bias.grad = {updater_net.u_fc0.bias.grad}')
params = dict(dict(updater_net.named_parameters()),**{'x':x,'y':y,'hidden':hidden},**dict(loss_net.named_parameters()))
make_dot(loss_val, params=params).render('loss_val', format='png')
</code></pre>

albanD: <p>Well you should not use <code>loss_net.named_parameters()</code> anymore. we override them. So you should “save” them before doing the first override. Like <code>params = list(loss_net.named_parameters())</code> before the for-loop.<br>
Then you can set them back (as leafs) when you want to make a new iteration. So maybe extract them as a state_dict and restore them with load_state_dict.</p>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="14" data-topic="70076">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Then you can set them back (as leafs) when you want to make a new iteration. So maybe extract them as a state_dict and restore them with load_state_dict.</p>
</blockquote>
</aside>
<p>I don’t think we should do that, if we do they would be used as non-leafs in the next iteration and won’t appear in the computation graph properly. It should be a chain of caused by the iterative use of <code>wt</code>.</p>
<p>Let me think about it…</p>
<p>I also thought we could have some special string like the word <code>param</code> in the field when we set it and thus we could loop through all the fields of the object that contain that special string.</p>

albanD: <p>Note that if the goal is to do learning through your optimizer step, the <a href="https://github.com/facebookresearch/higher">higher</a> library already implements this. And has a nice API for for all that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pinocchio: <p>thanks for that!</p>
<p>I’ve been playing around with the library but was wondering if it was possible to have a trainable step-size with that library. Is it possible?</p>
<p>I tried:</p>
<pre><code class="lang-auto">#
child_model = nn.Sequential(OrderedDict([
        ('conv1', nn.Conv2d(in_channels=3,out_channels=2,kernel_size=5)),
        ('relu1', nn.ReLU()),
        ('Flatten', Flatten()),
        ('fc', nn.Linear(in_features=28*28*2,out_features=10) )
    ]))
eta = nn.Sequential(OrderedDict([
    ('fc', nn.Linear(1,1)),
    ('sigmoid', nn.Sigmoid())
]))
inner_opt = torch.optim.Adam(child_model.parameters(), lr=eta)
meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_opt = torch.optim.Adam(meta_params, lr=1e-3)
</code></pre>
<p>but it failed with error:</p>
<pre><code class="lang-auto">Exception has occurred: TypeError
'&lt;=' not supported between instances of 'float' and 'Sequential'
</code></pre>

pinocchio: <p>Or even better the update rule be some sort of NN…</p>

pinocchio: <p>starting to think that going back to your settatrr approach might be better X’D…</p>
<aside class="quote quote-modified" data-post="1" data-topic="70396">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/implement-a-meta-trainable-step-size/70396">Implement a meta-trainable step size</a> 
  </div>
  <blockquote>
    I want to implement a (meta) trainable step size. I tried it with this post: 
<a href="https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17" class="onebox" target="_blank" rel="noopener">https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17</a> 
and with the higher library (<a href="https://github.com/facebookresearch/higher" class="inline-onebox-loading" rel="noopener nofollow ugc">https://github.com/facebookresearch/higher</a>) with no luck… 
I tried: 
eta = torch.tensor([0.5], requires_grad=True).view(1)
inner_opt = torch.optim.Adam(child_model.parameters(), lr=eta)
#meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_params = itertools.chain(child_…
  </blockquote>
</aside>


albanD: <p>You can simply replace the optimizer by a nn ?<br>
Or fill-in the <code>.grad</code> fiels with your nn’s result and then use a simple SGD to do the step.</p>