Weight regularization on selected filters

saluei: <p>At the start of each epoch, some of the filters were selected to be penalized with L2, if the filter selection procedure is in the training loop (loop on batch) everything is OK, but I want at the start of each epoch to select filters for regularization and then begin training , in this case I have popular error :<br>
“RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of …”, why THIS happens any help appreciated, I know computation graph generated dynamically. my code</p>
<pre><code class="lang-auto"> crt=nn.CrossEntropyLoss()
 for epochIdx in range(self.NumOfEpoch):
            reg_params = Variable( torch.FloatTensor(0), requires_grad=True)
            for n,x in self.model.named_parameters():
                if some condition meet:
                   reg_params = torch.cat([reg_params,x[[1,2],:,:,:].view(-1)]) #index 1 and 2 is sample

            #if all filter selected with this line and above for loop commented IT WORK !!!
            #reg_params = torch.cat([x.view(-1) for x in self.model.parameters()])

            for batchIdx, data in enumerate(self.trainloader, 0):
                inputs, labels = data
                inputs, labels = inputs.to(self.device), labels.to(self.device)

                self.model.zero_grad()
                features,output = self.model(inputs)
                l2_regularization = Reg_lambda * torch.norm(reg_params, 2)
                loss=crt(output,labels) + l2_regularization
                loss.backward()
                optimizer.step()
                #....
</code></pre>

InnovArul: <p>After every <code>optimizer.step()</code>, the params would be updated.  In this scenario,  the params used on L2 regularization would be outdated. You have to do the selection inside <code>for loop batch</code> I think.</p>

saluei: <p>thank you for your reply.<br>
yes, I know  when selection is done in batch for loop it work, i tested this matter before,<br>
However, due to performance/speed, I want to select the filters in epoch for loop,<br>
I do not get what is the meaning of “…the params used on L2 regularization would be outdated…”  pls explain it<br>
also when all filter is in selection (in epoch for loop) with this code It works!!!</p>
<pre><code class="lang-auto">reg_params = torch.cat([x.view(-1) for x in self.model.parameters()])
</code></pre>

InnovArul: <p>I guess I misunderstood the question.</p>
<aside class="quote no-group" data-username="saluei" data-post="1" data-topic="138545">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saluei/48/44594_2.png" class="avatar"> saluei:</div>
<blockquote>
<p><code>                if some condition meet:</code></p>
</blockquote>
</aside>
<p>What kind of condition is this? Does it involve any operation with parameters?</p>
<aside class="quote no-group" data-username="saluei" data-post="1" data-topic="138545">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saluei/48/44594_2.png" class="avatar"> saluei:</div>
<blockquote>
<pre><code class="lang-auto">            for n,x in self.model.named_parameters():
                if some condition meet:
                   reg_params = torch.cat([reg_params,x[[1,2],:,:,:].view(-1)]) #index 1 and 2 is sample
</code></pre>
</blockquote>
</aside>
<p>I am not sure if there is an issue with this code.<br>
Can you change it as follows and try?</p>
<pre><code class="lang-auto">reg_params = []
for n,x in self.model.named_parameters():
    if some condition meet:
        reg_params.append(reg_params,x[[1,2],:,:,:].view(-1)) #index 1 and 2 is sample

reg_params = torrch.cat(reg_params, dim=0)
</code></pre>
<p>It will be great if you could provide a compact runnable code to see whats the issue.</p>

saluei: <p>No condition does not have any parameters and operation.<br>
Your suggestion does not work and has the error “Trying to backward through the graph a second time (or directly access saved tensors…”<br>
I change the code to this and it works but I do not know why this works??<br>
this line :</p>
<pre><code class="lang-auto">reg_params = torch.cat([reg_params,x[[1,2],:,:,:].view(-1)])
</code></pre>
<p>changed to</p>
<pre><code class="lang-auto">temp =[1,2]
for i in temp:
      reg_params = torch.cat([reg_params,x[i,:,:,:].view(-1)])
....
</code></pre>
<p>Thank you InnovArul</p>