Can `register_forward_pre_hook` modify module weight, and is modified weight learnable?

username: shipley223
cooked: <p>In Pytorch Doc It says <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_pre_hook" rel="noopener nofollow ugc">The hook can modify the input</a>, but there no doucument about module weight, my question is can  <code>register_forward_pre_hook modify</code> module weight? and is the modified weight learnable?<br>
I Noticed that <a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/utils/weight_norm.py" rel="noopener nofollow ugc">weight_norm</a> use this method to modify weight:</p>
<ol>
<li>delete original weight <code>w_original</code> name in <code>_parameters</code></li>
<li>register new weight <code>w_new</code> using <code>register_paramerter</code></li>
<li>using <code>register_forward_pre_hook</code> to register a hook function which set module <code>w_new</code>  to ``w_original`</li>
</ol>
<p>I want to know after these three steps if the <code>w_original</code> point to <code>w_new</code>, and <code>w_new</code> can be updated using SGD?</p>
<p>I try to simulate quantization using <code>register_forward_pre_hook</code> and change <code>compute_weight</code> in <a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/utils/weight_norm.py" rel="noopener nofollow ugc">weight_norm</a><br>
from</p>
<pre data-code-wrap="python"><code class="lang-python">def compute_weight(self, module: Module) -&gt; Any:
        g = getattr(module, self.name + '_g')
        v = getattr(module, self.name + '_v')
        return _weight_norm(v, g, self.dim)
</code></pre>
<p>to</p>
<pre><code class="lang-auto"># i want to simulate weight quantization Q10
def compute_weight(self, module: Module) -&gt; Any:
        w = getattr(module, self.name + 'quantized')
        return torch.round(w * 2**10) / 2**10
</code></pre>
<p>but the loss remains constant, Could someone help me to figure it out?<br>
Thanks a lot~</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="shipley223" data-post="1" data-topic="202711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shipley223/48/69271_2.png" class="avatar"> shipley223:</div>
<blockquote>
<pre><code class="lang-auto">        return torch.round(w * 2**10) / 2**10
</code></pre>
</blockquote>
</aside>
<p>Isn’t <code>torch.round</code> non-differentiable?</p>

username: ptrblck
cooked: <p><code>torch.round</code> is technically differentiable, but will return zeros, so it’s not really usefully differentiable:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(5, requires_grad = True)
out = torch.round(x)
out.mean().backward()
print(x.grad)
# tensor([0., 0., 0., 0., 0.])
</code></pre>
<p>The derivative is defined in <a href="https://github.com/pytorch/pytorch/blob/b522e65056d52c4da08ed8f5a8799e36738889aa/tools/autograd/derivatives.yaml#L1440-L1442"><code>derivatives.yaml</code></a> as:</p>
<pre data-code-wrap="python"><code class="lang-python">- name: round(Tensor self) -&gt; Tensor
  self: zeros_like(grad)
  result: auto_element_wise
</code></pre>

username: shipley223
cooked: <p>I want to convert trained model’s weight from float32 to Q10 fix-point number, but the performance will reduce about 20%, could you give some advice on how to take quantization error(weight from float32 to Q10 fixed point number) into consideration?</p>