How to put constrain while minimising loss function

Sunil_Sharma1: <p>I am trying to minimise the mse loss with constrain loss but constrain was increasing instead of decreasing.then i tried to only minimise constrain then it throw following error.</p>
<pre><code class="lang-auto">
class Edge_Detector(nn.Module):
  def __init__(self,kernel_size,padding):
    torch.manual_seed(1)
    super(Edge_Detector,self).__init__()
    self.sobelx=nn.Conv2d(1,1,kernel_size=kernel_size,padding=padding,bias=False)
    self.relu=nn.ReLU()
    self.sobely=nn.Conv2d(1,1,kernel_size=kernel_size,padding=padding,bias=False)

  def forward(self,x):
    x1=self.sobelx(x)
    x2=self.sobely(x)
    x=self.relu(x1+x2)
    return x

  def loss(self,x,y):
    x=x.view(x.size(0),-1)
    y=y.view(y.size(0),-1).float()
    sobelx=self.sobelx.weight.data.squeeze().squeeze()
    sobely=self.sobely.weight.data.squeeze().squeeze()
    loss_mse=nn.MSELoss()(x,y)
    loss_constrain=torch.matmul(sobelx,sobely.transpose(0,1)).trace()
    #print('mse_loss :  ',loss_mse)
    #print('constrain_loss  : ',loss_constrain)
    #total_loss=loss_mse+loss_constrain
    return loss_constrain

#Error Message:
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-67-28b5b5719682&gt; in &lt;module&gt;()
----&gt; 1 learn.fit_one_cycle(15, 5e-2)               #training for 4 epochs with lr=1e-3

13 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--&gt; 132         allow_unreachable=True)  # allow_unreachable flag
    133 
    134 

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

</code></pre>

ChickenTarm: <p>I have never seen a constraint like that, but from that loss_constraint it appears that you want sobelx and sobely weights to be orthagonal?</p>
<p>My guess is that this isn’t allowed because the gradients would need to be based off some output as opposed to the weights themselves.</p>
<p>What you might be able to do is run some input through sobelx and sobely and then constrain their outputs to be orthogonal</p>

KFrank: <p>Hi Sunil!</p>
<aside class="quote no-group quote-modified" data-username="Sunil_Sharma1" data-post="1" data-topic="109791" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunil_sharma1/48/15587_2.png" class="avatar"> Sunil_Sharma1:</div>
<blockquote>
<p>I am trying to minimise the mse loss with constrain loss but constrain was increasing instead of decreasing.then i tried to only minimise constrain then it throw following error.</p>
<pre><code class="lang-auto">    self.sobelx=nn.Conv2d(1,1,kernel_size=kernel_size,padding=padding,bias=False)
    self.relu=nn.ReLU()
    self.sobely=nn.Conv2d(1,1,kernel_size=kernel_size,padding=padding,bias=False)
...
    sobelx=self.sobelx.weight.data.squeeze().squeeze()
    sobely=self.sobely.weight.data.squeeze().squeeze()
    loss_constrain=torch.matmul(sobelx,sobely.transpose(0,1)).trace()
...
#Error Message:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
</blockquote>
</aside>
<p>Your use of <code>.data</code> “breaks the computation graph,” so that no gradients are<br>
backpropagated through <code>loss_constrain</code>.  Therefore, when you only minimize<br>
<code>loss_constrain</code> you get the reported error.  When you also minimize<br>
<code>loss_mse</code> (assuming that <code>loss()</code> returns <code>total_loss</code>) you do have gradients<br>
that connect <code>total_loss</code> to your model parameters, so the error goes away.</p>
<p>However, <code>loss_constrain</code> is still disconnected from the computation graph,<br>
so its gradients are not calculated and make no contribution to the gradients<br>
calculated for <code>total_loss</code>.  Therefore your gradient-descent optimization<br>
knows nothing about <code>loss_constrain</code> and there will be no tendency for<br>
<code>loss_constrain</code> to be made smaller.</p>
<p>Also note that even if you remove <code>.data</code>, I believe that there is nothing that<br>
prevents <code>loss_constrain</code> (the trace of the product of your two convolution<br>
kernels) from becoming large and negative.  So gradient descent will, roughly<br>
speaking, ultimately drive one kernel off to <code>+inf</code> and the other off to <code>-inf</code> so<br>
that <code>loss_constrain</code> becomes <code>-inf</code> (and dominates <code>total_loss</code>).</p>
<p>I don’t really follow what you are trying to do with <code>loss_constrain</code>, but<br>
something like:</p>
<pre data-code-wrap="python"><code class="lang-python">total_loss = loss_mse + wt_constrain * loss_constrain**2
</code></pre>
<p>would at least prevent the <code>-inf</code> problem.  (And, yes, I would include the<br>
weighting factor <code>wt_constrain</code> so that you can adjust how strongly<br>
<code>loss_constrain</code> contributes to your overall loss.)</p>
<p>(One final note:  Putting <code>torch.manual_seed(1)</code> in the constructor for<br>
<code>Edge_Detector</code> is not a good idea, as this sets the seed for the “global”<br>
random number generator, affecting any other random-number calls, and<br>
should you construct more than one <code>Edge_Detector</code>, the seed will be<br>
reset.  This could work for your use case, but is likely to become fragile as<br>
you add more code to your system.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

Sunil_Sharma1: <p>thanks  it ran without any error.but<br>
I want to learn the kernels given like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d27e1758466c006821e03e33eb316099f7e5ff4.png" alt="sobel" data-base62-sha1="8J0zjVTKztdn0KZ4xIdOHFHLbJa" width="234" height="306"><br>
or similar pattern like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f4aff3a8c8aa2c2dac27a1828cb9a4ab4f600f.png" alt="gxy" data-base62-sha1="uf2Z9LFq7VGRjQFwSwibBZNHFeL" width="454" height="220"><br>
but after optimising i am getting like this.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22679ab8fb5e32e567dfda3d333233c46969a9ab.png" alt="soblxy" data-base62-sha1="4UmcJqCedTvbeP926fGmfi29uEH" width="452" height="240"></p>
<p>here is my loss function</p>
<pre><code class="lang-auto">def loss(self,x,y):
    x=x.view(x.size(0),-1)
    y=y.view(y.size(0),-1).float()
    sobelx=self.sobelx.weight.squeeze().squeeze()
    sobely=self.sobely.weight.squeeze().squeeze()
    loss_mse=nn.MSELoss()(x,y)
    loss_constrain=torch.matmul(sobelx,sobely.transpose(0,1)).trace()
    w=.8
    total_loss=(1-w)*loss_mse + w*loss_constrain**2
    return total_loss
</code></pre>
<p>i want to know  how i can learn my desired filters?</p>

KFrank: <p>Hi Sunil!</p>
<aside class="quote no-group" data-username="Sunil_Sharma1" data-post="4" data-topic="109791" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunil_sharma1/48/15587_2.png" class="avatar"> Sunil_Sharma1:</div>
<blockquote>
<p>I want to learn the kernels given like this</p>
</blockquote>
</aside>
<p>If you want your kernels to be equal to the non-textual G_x and G_y you<br>
gave, why don’t you just set them to be equal, rather than try to “learn”<br>
them?</p>
<p>In a related point, it is true that your G_x and G_y satisfy your desired<br>
constraint, and, indeed, <code>loss_constrain</code> takes on its minimum value of<br>
zero with them.  But so do many, many other pairs of 3x3 matrices.  So<br>
there is no reason that training with <code>loss_constrain</code> will result in kernels<br>
that are close to G_x and G_y.</p>
<p>Best.</p>
<p>K. Frank</p>