Why my model params don't change by back propagation?

username: yangxi86
cooked: <p>I tried to made a single waveshaper node referred from <a href="https://benhayes.net/projects/nws/" rel="noopener nofollow ugc">this work</a>, in which input is transformed by value table lookup and interpolation:</p>
<pre><code class="lang-auto">import torch

class SampledWaveshaper(torch.nn.Module):
    def __init__(self, table_min: float, table_max: float, table_size: int):
        super().__init__()
        self.table_min = table_min
        self.table_max = table_max
        self.table_size = table_size
        self.lookup_table = torch.nn.Parameter(torch.linspace(table_min, table_max, table_size, requires_grad=True))


    def forward(self, x):
        fidx = self.table_size * (x - self.table_min) / (self.table_max - self.table_min)
        ilow = torch.floor(fidx).long()
        ilow[ilow &lt; 0] = 0
        ilow[ilow &gt;= self.table_size] = self.table_size-1

        ihigh = torch.ceil(fidx).long()
        ihigh[ihigh &lt; 0] = 0
        ihigh[ihigh &gt;= self.table_size] = self.table_size-1
        
        val1 = torch.stack([self.lookup_table[i] for i in ilow])
        val2 = torch.stack([self.lookup_table[i] for i in ihigh])

        r = fidx - ilow
        result = val1 + (val2-val1) * r
        return result
</code></pre>
<p>And very simple training code:</p>
<pre><code class="lang-auto"># do training
# do training
cri = torch.nn.MSELoss()
opt = torch.optim.SGD(shaper.parameters(), lr=1e-6)

input = torch.zeros(args.batch_sz)
refout = torch.zeros(args.batch_sz)

for iter in range(args.n_iter):
    # assemble input and reference output
    for i in range(args.batch_sz):
        while True:
            i_data = random.randrange(len(all_data))
            curr_in = float(all_data[i_data][0])
            if shaper.table_min &lt;= curr_in and curr_in &lt;= shaper.table_max:
                input[i] = curr_in
                refout[i] = float(all_data[i_data][1])
                break
        
        
    output = shaper.forward(input)
    loss = cri(output, refout)
    opt.zero_grad()
    loss.backward()
    opt.step()
</code></pre>
<p>It seems <code>input</code>, <code>output</code> and <code>refout</code> all looks good, but the <code>lookup_table</code> in model is not changed at all.</p>

username: KFrank
cooked: <p>Hi Xi!</p>
<aside class="quote no-group" data-username="yangxi86" data-post="1" data-topic="160605" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/22d042/48.png" class="avatar"> yangxi86:</div>
<blockquote>
<p>the <code>lookup_table</code> in model is not changed at all.</p>
</blockquote>
</aside>
<p>Your <code>SampledWaveshaper</code> model works for me when I give it some<br>
simple input data.</p>
<p>Note, the learning rate (<code>lr=1e-6</code>) that you pass to the <code>SGD</code> optimizer<br>
is quite small.  Is it possible that what would have been changes to<br>
<code>shaper.lookup_table</code> are underflowing to zero because of the small<br>
learning rate (or maybe are just small enough that you don’t notice<br>
them)?</p>
<p>Here is a simple test script that shows sensible gradients and non-zero<br>
changes to <code>shaper.lookup_table</code> after calling <code>opt.step()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; class SampledWaveshaper(torch.nn.Module):
...     def __init__(self, table_min: float, table_max: float, table_size: int):
...         super().__init__()
...         self.table_min = table_min
...         self.table_max = table_max
...         self.table_size = table_size
...         self.lookup_table = torch.nn.Parameter(torch.linspace(table_min, table_max, table_size, requires_grad=True))
...
...     def forward(self, x):
...         fidx = self.table_size * (x - self.table_min) / (self.table_max - self.table_min)
...         ilow = torch.floor(fidx).long()
...         ilow[ilow &lt; 0] = 0
...         ilow[ilow &gt;= self.table_size] = self.table_size-1
...
...         ihigh = torch.ceil(fidx).long()
...         ihigh[ihigh &lt; 0] = 0
...         ihigh[ihigh &gt;= self.table_size] = self.table_size-1
...
...         val1 = torch.stack([self.lookup_table[i] for i in ilow])
...         val2 = torch.stack([self.lookup_table[i] for i in ihigh])
...
...         r = fidx - ilow
...         result = val1 + (val2-val1) * r
...         return result
...
&gt;&gt;&gt; shaper = SampledWaveshaper (1.0, 5.0, 5)
&gt;&gt;&gt;
&gt;&gt;&gt; opt = torch.optim.SGD(shaper.parameters(), lr=1e-6)   # this learning rate is small
&gt;&gt;&gt;
&gt;&gt;&gt; input = 3 * torch.ones (3)
&gt;&gt;&gt; refout = 3 * torch.ones (3)
&gt;&gt;&gt; output = shaper.forward (input)   # shaper (input) is more standard
&gt;&gt;&gt;
&gt;&gt;&gt; opt.zero_grad()
&gt;&gt;&gt; torch.nn.MSELoss() (output, refout).backward()
&gt;&gt;&gt;
&gt;&gt;&gt; shaper.lookup_table.grad   # grad is reasonable and of order one
tensor([0.0000, 0.0000, 0.5000, 0.5000, 0.0000])
&gt;&gt;&gt;
&gt;&gt;&gt; param_before = shaper.lookup_table.clone().detach()
&gt;&gt;&gt; opt.step()
&gt;&gt;&gt; param_after = shaper.lookup_table.clone().detach()
&gt;&gt;&gt;
&gt;&gt;&gt; param_after - param_before   # difference is close to float precision due to small lr
tensor([ 0.0000e+00,  0.0000e+00, -4.7684e-07, -4.7684e-07,  0.0000e+00])
</code></pre>
<p>If you are unable to resolve your issue, please post a minimal, simplified,<br>
runnable script that illustrates your issue together with the output that it<br>
produces.</p>
<p>Best.</p>
<p>K. Frank</p>

username: yangxi86
cooked: <p>Hi KFrank! Thanks for your suggestion. I’ve almost forgot that 1e-6 is very near to the precision limit of 32-bit float.</p>