Freezing Individual Weights

rahulvigneswaran: <p>I am trying out a pytorch implementation of Lottery Ticket Hypothesis. For that, I want to freeze the weights in a model that are zero. Is the following a correct way to implement it ?</p>
<pre><code class="lang-auto">for name, p in model.named_parameters():
            if 'weight' in name:
                tensor = p.data.cpu().numpy()
                grad_tensor = p.grad.data.cpu().numpy()
                grad_tensor = np.where(tensor == 0, 0, grad_tensor)
                p.grad.data = torch.from_numpy(grad_tensor).to(device)
</code></pre>

dejanbatanjac: <p>The question is tricky. First the <a href="https://arxiv.org/abs/1803.03635" rel="nofollow noopener">paper</a> is suspicious… At least for me.</p>
<p>I just fast-forward it. They seam to optimize the computation to reduce the heat…</p>
<p>Definitely, not the memory because the shape of the tensors will be the same.<br>
The original idea of the parameters is they should not be read only.</p>
<p>If you freeze weights then how can you learn?<br>
If huge amount of your weight values are 0 you haven’t initialized your layer well, or mean and std are not normalized going through the layers.<br>
I would ignore this paper.</p>
<p>If from some reason you would still like to deal with the paper (say you would like to implement it):</p>
<ul>
<li>
<p>Try using <code>model.parameters()</code> instead of <code>model.named_parametrers()</code>, since named parameters are subset of parameters.</p>
</li>
<li>
<p><code>.numpy()</code> would convert tensor to numpy array, so don’t set that to a tensor. In your case <code>tensor</code> and <code>grad_tensor</code> should be tensors with the same shape.</p>
</li>
<li>
<p>use <code>torch.from_numpy(arr)</code> if you have numpy arrays and you would like to convert them to tensors.</p>
</li>
<li>
<p>Although CUDA code runs on both CPU and GPU, I would not hardcode <code>.cpu()</code> or <code>.gpu()</code>, instead always use <code>.to(device)</code>, unless you are positive you will only run your code on GPU. To detect the device you can use this:</p>
</li>
</ul>
<pre><code class="lang-auto">device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
</code></pre>

rahulvigneswaran: <p>I don’t find the paper suspicious. I believe it conveys a nice property of, “<strong>Almost all the time, if you give me a network, I can point you to a subnetwork that is capable of giving comparable accuracies</strong>”.</p>
<p>It is true that even though we call it a compression, it is still allocated in the memory as Zero and the shape is retained but it is <strong>more of an implementation flaw rather than the result it is trying to convey</strong>. As I mentioned earlier the result, “<strong>Almost all the time, if you give me a network, I can point you to a subnetwork that is capable of giving comparable accuracies</strong>” is loud and clear.</p>
<p>Also, by <strong>freezing weights</strong>, we are freezing it to zero, potentially removing that neuron, hence it starts to act like a new network altogether.</p>
<p>I think it’s a fascinating paper and the following follow-up papers of the Lottery Ticket Hypothesis will help you understand my fascination.</p>
<ol>
<li><a href="https://arxiv.org/abs/1903.01611" rel="nofollow noopener">Stabilizing the Lottery Ticket Hypothesis</a></li>
<li><a href="https://eng.uber.com/deconstructing-lottery-tickets/" rel="nofollow noopener">Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask</a></li>
</ol>

rahulvigneswaran: <p>The answer to this question was answered on <a href="https://stackoverflow.com/questions/58145727/freezing-individual-weights-in-pytorch" rel="nofollow noopener">stackoverflow</a> by a user named <a href="https://stackoverflow.com/users/story/2790047" rel="nofollow noopener">jodag</a>. I have modified and added it here.</p>
<p><strong>SOLUTION:</strong></p>
<p>What you have seems like it would work provided you did it after  <code>loss.backward()</code>  and before  <code>optimizer.step()</code>  (referring to the common usage for these variable names). That said, it seems a bit convoluted. Also, if your weights are floating-point values then <a href="https://stackoverflow.com/questions/588004/is-floating-point-math-broken" rel="nofollow noopener">comparing them to exactly zero is probably a bad idea</a>, we could introduce an epsilon to account for this.</p>
<p>The modified code :</p>
<pre><code class="lang-python">EPS = 1e-6       #Changes
for name, p in model.named_parameters():
            if 'weight' in name:
                tensor = p.data.cpu().numpy()
                grad_tensor = p.grad.data.cpu().numpy()
                grad_tensor = np.where(tensor &lt; EPS, 0, grad_tensor)     #Changes
                p.grad.data = torch.from_numpy(grad_tensor).to(device)
</code></pre>