Custom layer gets same weights in every training iterations

joshua2: <p>Hello, everyone<br>
I want to make a custom regularization layer with Pytorch but something is wrong to my regularization layer because the loss output is all same when training.<br>
The real problem is that <strong>I found out myloss gets same net.parameters() in every training process but, I do not know why it gets same weight parameters even if I give updated network</strong></p>
<p>My custom layer is like below</p>
<blockquote>
<p>class bipolar_loss(nn.Module):</p>
</blockquote>
<pre><code>def __init__(self, lambd=5e-7):
    super(bipolar_loss, self).__init__()
    self.lambd = lambd

def forward(self, net):

    loss = 0
    for param in net.parameters():
        # Only for weight parameters
        if len(param.size()) == 4:
            loss += (1 - torch.pow(param, 2)).sum()
    return loss * self.lambd
</code></pre>
<p>And I use this layer in main.py like below</p>
<blockquote>
<p>criterion = nn.CrossEntropyLoss().cuda()<br>
loss_func = bipolar_loss().cuda()</p>
</blockquote>
<p>In the training process, what I want to do is to add two losses like L2 regularization so I add two losses in every training iterations</p>
<blockquote>
<pre><code>for batch_idx, (inputs, targets) in enumerate(trainloader):
    inputs, targets = inputs.to(device), targets.to(device)
    optimizer.zero_grad()
    
    # forward pass to get output / logits
    outputs = net(inputs)

    # calculate loss *default is cross entropy loss
    loss = criterion(outputs, targets)
    myloss = loss_func(net)

    loss = loss + myloss

    # getting gradients parameters
    loss.backward()
    
    # updating parameters
    optimizer.step()
    
    train_loss += loss.item()
    _, predicted = outputs.max(1)
    total += targets.size(0)
    correct += predicted.eq(targets).sum().item()

epoch_loss = train_loss/(batch_idx+1)
epoch_acc = 100.*correct/total
</code></pre>
</blockquote>
<p>However, I got same loss when printing the output of myloss</p>
<blockquote>
<p>iteration 0:<br>
loss:  tensor(2.2010, device=‘cuda:6’, grad_fn=NllLossBackward)<br>
<strong>myloss:  tensor(2.3100, device=‘cuda:6’, grad_fn=MulBackward0)</strong><br>
total_loss: tensor(4.5111, device=‘cuda:6’, grad_fn=AddBackward0)</p>
</blockquote>
<blockquote>
<p>iteration 1:<br>
loss:  tensor(2.2096, device=‘cuda:6’, grad_fn=NllLossBackward)<br>
<strong>myloss:  tensor(2.3100, device=‘cuda:6’, grad_fn=MulBackward0)</strong><br>
total_loss: tensor(4.5196, device=‘cuda:6’, grad_fn=AddBackward0)</p>
</blockquote>
<p>myloss is all same in every iterations and epochs…</p>

ayalaa2: <p>Don’t use in-place operations like “+=”. They don’t play nicely with autograd, you can read more <a href="https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd" rel="nofollow noopener">here</a>.</p>
<p>So instead of:<br>
<code>loss += (1 - torch.pow(param, 2)).sum()</code><br>
use:<br>
<code>loss = loss + (1 - torch.pow(param, 2)).sum()</code></p>
<p>Try this change and see if it affects what you’re seeing.</p>

joshua2: <p>Thank for your kind advice!<br>
I checked the result changing the in-place operation to loss = loss + ~, but the result is same (same myloss). The critical problem is that the layer gets same parameters in every training process.<br>
But of course, my goal is to make myloss correctly.</p>

ayalaa2: <p>Hmm, that’s interesting. It might be that PyTorch isn’t recording the operations done on it because <code>loss</code> inside <code>bipolar_loss</code> is a python scalar.</p>
<p>Instead of:<br>
<code>loss = 0</code><br>
try:<br>
<code>loss = torch.tensor(0.0, requires_grad=True)</code></p>
<p>Let me know if that does something different.</p>

joshua2: <p>oh… I got error like this</p>
<blockquote>
<p>File “main.py”, line 134, in train<br>
loss.backward()<br>
File “/home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 98, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: Function AddBackward0 returned an invalid gradient at index 0 - expected type TensorOptions(dtype=float, device=cuda:6, layout=Strided, requires_grad=false) but got TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false) (validate_outputs at /opt/conda/conda-bld/pytorch_1587428094786/work/torch/csrc/autograd/engine.cpp:484)<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7feb06c6eb5e in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0x2ae2834 (0x7feb30932834 in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#2:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr<a>torch::autograd::GraphTask</a>&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0x548 (0x7feb30934368 in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span> torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;, bool) + 0x3d2 (0x7feb309362f2 in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span> torch::autograd::Engine::thread_init(int) + 0x39 (0x7feb3092e969 in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7feb33c6f9f8 in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xc819d (0x7feb366c719d in /home/sangwooj/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/lib/…/…/…/…/./libstdc++.so.6)<br>
frame <span class="hashtag-raw">#7:</span>  + 0x76ba (0x7feb4ecb56ba in /lib/x86_64-linux-gnu/libpthread.so.0)<br>
frame <span class="hashtag-raw">#8:</span> clone + 0x6d (0x7feb4e9eb41d in /lib/x86_64-linux-gnu/libc.so.6)</p>
</blockquote>

ayalaa2: <p>Okay that’s weird, give me a sec I’m going to see if I can reproduce it locally…</p>

ayalaa2: <p>Okay ignore the second suggestion I made. I think it resulted in that error when using CUDA.</p>
<p>I think your loss function is actually working, but the gradients as a result are incredibly small. In my tests I’m seeing gradients as small as 1e-8. Which when using a learning rate such as 1e-2, the update to your weights are incredibly small. So your loss is appearing to be unmoving, but it actually is. Set your <code>lambd</code> to some larger number for testing, such as 10.0. I think you’ll see a change in your loss now.</p>

joshua2: <p>Thanks for your comment!<br>
Does myloss output have the same value in your code? In my training process, myloss value is all same</p>

ayalaa2: <p>I made a simple model that has one layer (linear, so I adjusted your if condition) and the only objective function was yours. I did see that it was unchanging at 0.005 after an optimization step until I bumped up lambd, then I saw that it was changing.</p>
<p>Even with bumping your lambd value up you don’t see a change in your myloss term?</p>

joshua2: <p>You are right… If I change the lambd to 0.005, then myloss is changed very slightly in every iterations.<br>
Thanks for your help. But, the result is not what I wanted <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"> because it is too large comparing the difference between before and after (like iteration1: myloss 23083.1270, iteration 2: myloss 23083.1640)<br>
I think I have to consider why it happens like that. Thank you!</p>

ayalaa2: <p>Keep in mind that loss values are completely arbitrary. For some loss functions, a value of 10 is small and for other loss functions, a value of 10,000 is small.</p>
<p>Since your loss term is the sum of every parameter value, we can expect your loss term to grow with the number of parameters. So it’s fine to have a loss value of say 20,000, the important thing is that your gradients aren’t insane.</p>
<p>If you prefer to have a smaller value, perhaps consider taking the average per parameter group. (so mean() instead of sum()). If you work out the math, we’ll find that the gradients have the same proportions in respect to each parameter value within the same parameter groups, just at different scales. So it’s almost the same for all intents and purposes. I would suggest taking the average because a “low” value will be the same regardless of size of model.</p>
<p>Also keep in mind that when you have 2 or more loss objectives that they might conflict with eachother! As in, reducing loss1 may result in an increase of loss2.</p>