Problem with zeroing gradients

username: ciw
cooked: <p>Hi, I have a problem which is bothering me for a week, any suggestion would be appreciated.</p>
<p>The problem is, when I tried to zeroing out part of the gradients, the corresponding weights are not frozen as expected, these weights are moving in a very small order similar to a truncation error.</p>
<p>In my application, I need to zero out part of the gradients under specific conditions. Here’s an example code to reproduce the error.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/ciw-ws/191019_zero_gradients" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/33036392?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/ciw-ws/191019_zero_gradients" target="_blank" rel="nofollow noopener">ciw-ws/191019_zero_gradients</a></h3>

<p>example code of zeroing gradients. Contribute to ciw-ws/191019_zero_gradients development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The code is modified from the following pytorch example.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/imagenet/main.py</a></h4>
<pre><code class="lang-py">import argparse
import os
import random
import shutil
import time
import warnings

import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.distributed as dist
import torch.optim
import torch.multiprocessing as mp
import torch.utils.data
import torch.utils.data.distributed
import torchvision.transforms as transforms
import torchvision.datasets as datasets
import torchvision.models as models

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The 3 ways I use to zero out gradients, the error is still there.</p>
<pre><code class="lang-auto">model.conv1.weight.grad *= 0
model.conv1.weight.grad.fill_(0)
model.conv1.weight.grad.data.zero_()
</code></pre>
<p>The error is small so it does not affect the performance much, but I’d like to understand how does this happen. Thanks a lot.</p>

username: tom
cooked: <p>If you use momentum or weight decay (from the optimizer), you will get changing parameters even when the grad has the numerical value 0.<br>
Optimizers do special case None gradients to mean “do nothing”, but that isn’t available for parts of a parameter, but in your example <code>model.conv1.weight.grad = None</code> would do the trick.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ciw
cooked: <p>Thanks Thomas.</p>
<p>Now I understand what is happening.</p>