How to restrict the neural network gradient value w.r.t. some input

ElleryL: <p>Hi; I’m interested to learn a function <code>NN(x1,x2)</code> such that <code>derivative of NN(x1,x2)</code> w.r.t. <code>x2</code> to be positive.</p>
<p>In my code; I have done <code>x1.shape = (N,D)</code> and <code>x2.shape = (N,1)</code> where <code>N</code> is the number of data points.  <code>X = torch.cat([x1,x2],dim=-1)</code> then do  <code>NN(X)</code>.</p>
<p>Now, I only want to restrict the derivative of <code>x2</code> to be positive and this makes sense as <code>x2</code> is only one dimension while keep ‘x1’ to be free but I don’t want to train two different neural network here.</p>
<p>I’m wondering what’s the proper and safe way of doing this ?</p>

albanD: <p>Hi,</p>
<p>How do you enforce that the derivative for <code>X2</code> is positive?</p>

ElleryL: <p>hi; that’s my question; I’m wondering if its possible</p>

albanD: <p>Well the derivative is what it is. You cannot really change its value.<br>
It’s like if you have a model, how do you enforce that the output is always positive?</p>

ElleryL: <p>I’m thinking about adding regularization term such that for each X1, max(− ∂ NN (X1 ,X2 )  /  ∂ X2, 0)</p>
<p>but I’m not sure how to do this properly</p>

albanD: <p>You can do this as follows:</p>
<pre><code class="lang-python">X1, X2 = some tensors
loss_weighting = some constant to weight the different losses

X2.requires_grad_()

task_loss = NN(X1, X2)

gX2 = torch.autograd.grad(task_loss, X2, create_graph=True)[0]
penalty_loss = torch.max(-gX2, 0).sum() # other things than sum could be used here...

loss = task_loss + loss_weighting * penalty_loss

optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>

ElleryL: <p>Thanks for the suggestion. Just few questions</p>
<p>(1) <code>loss = task_loss + loss_weighting * penalty_loss</code> and then <code>loss.backward()</code>. Why do I backward on penalty_loss ? Since <code>torch.max(-gX2, 0).sum() </code> already gives my the derivative of <code>X2</code>; Isn’t  <code>loss.backward()</code> gonna compute the second derivative of <code>X2</code> ?</p>

albanD: <p><code>torch.max(-gX2, 0).sum()</code> gives you the penalty you want to apply.<br>
I assumed that you wanted to try and apply it with gradient descent, the same way you learn your original loss.<br>
So you want the derivative of this new penalty (and yes this is a second derivative-like thing) wrt your weights.</p>

ElleryL: <p>Oh; I see is loss_weighting lagrange multiplier ?</p>
<p>minimize taks_loss subjected to ∂ NN (X1 ,X2 ) / ∂ X2 &gt; 0</p>
<p>equivalent to minimize taks_loss subjected to (- ∂ NN (X1 ,X2 ) / ∂ X2 &lt; 0)</p>
<p>so it becomes taks_loss +  loss_weighting * ( - ∂ NN (X1 ,X2 ) / ∂ X2 ) ?</p>
<p>Just one more question in the code it looks like</p>
<pre><code class="lang-auto">
for iters in range(total_runs):
    # some code here
    torch.autograd.grad(task_loss, X2, create_graph=True)[0] # does it takes lots of memory ?
</code></pre>
<p>since we are creating graph each iteration; does it gonna take massive memory as the training goes on ? Does the algorithm free graph after <code>optimizer.step()</code> just like regular optimization ?</p>

albanD: <p>It will use more memory as it does more computations but the graph will be deleted as soon as its not needed (the beginning of the next iteration).<br>
So you don’t need to worry about leaking memory <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ElleryL: <p>Thanks; so I can think it takes O(1) memory in term of number of train iterations</p>

albanD: <p>Yes O(1) in terms of the number of iterations.</p>

ElleryL: <p>Thanks for the help!</p>

ElleryL: <p>hi just one more thing to follow. how do I use <code>torch.autograd.grad(y,X2)</code> when my <code>y = NN(X1,X2)</code> is a matrix <code>NxD</code> where N is number of points and D is dimension ? I did<code> torch.autograd.grad(y, X2, create_graph=True,grad_outputs=torch.ones_like(y))[0]</code> it returns tensor with shape <code>(N,)</code> (it take derivative of the sum over dimension); but I’m more expecting <code>(N,D)</code> that take derivative of each dimension and keep derivative of each dimension to be positive</p>
<p>My toy example is</p>
<pre><code class="lang-auto">
x = torch.randn(5,1) # my X1 above
z = torch.ones(5,1) # my X2 above
z.requires_grad = True
f = nn.Linear(2,3)
x = torch.cat([x,z],dim=-1)
y = f(x)
torch.autograd.grad(y, z, create_graph=True,grad_outputs=torch.ones(y.size()))[0] # gives 5x1 but expecting 5x3

</code></pre>
<p>To be more specifically; for example for a single data</p>
<p>I have a vector-valued function such that</p>
<p>f(x,z) = [y1,y2,y3 ] =[x+3z,x+4z,x+5z]</p>
<p>Then, df/dz = [3,4,5]; but the autograd gives me the sum [12]</p>
<p>so I guess the autograd computes as</p>
<p><code>torch.ones(y.shape) @ [ [dy1 / dx, dy1 / dz] , [dy2 / dx, dy2 / dz] , [dy2 / dx, dy2 / dz] ]   = [ [1,1,1] ] @ [ [1,3],[1,4],[1,5] ]</code></p>
<p>Is there anyway that not compute this final matrix multiplication?</p>
<p>The naive way would be</p>
<pre><code class="lang-auto">J = []
for i in range(D):
    out = torch.zeros(1,D)
    out[0][i] = 1
    j = torch.autograd.grad(y, x, create_graph=True,grad_outputs=out)[0]
    J.append(j[0])
J = torch.stack(J)
dy_dz = J[:,-1]
</code></pre>
<p>This is just not gonna work as I can’t loop high dimension every iteration and I don’t think its reasonable as I only want the dy_dz (The last column of Jacobean ) and required to loop all dimension for full Jacobean</p>

albanD: <p>Hi,</p>
<p>Unfortunately the autograd does not compute Jacobians, but vector Jacobian products.<br>
Note that here I did not give the output of NN to the <code>.grad()</code> but the <code>loss</code> itself. There are two main reasons here:</p>
<ul>
<li>I would argue that the gradient you want to enforce is not on the Jacobian of the function but the gradient of the loss wrt the input.</li>
<li>The loss is scalar, so a single backward pass actually gives the full Jacobian (gradients) and you can do the penalty directly without having to do a for-loop of <code>.grad()</code>
</li>
</ul>

ElleryL: <p>Hi</p>
<p>So I’m bit of confused; because my main plan is to make neural network NN(X1,X2); (where X2 is a scalar, and X1 is a vector. and the output of NN(X1,X2) could be any dimension. ) to be monotonic non-decreasing w.r.t. X2; that it to say NN(X1,X2+ epsilon) &gt;= NN(X1,X2) for any epsilon &gt; 0 for all output dimension. That’s why I’m restricting its derivative to be non-negative. However, this seems doesn’t gonna work in pytorch ?</p>

albanD: <p>Ho,</p>
<p>I did not understood your goal.<br>
So yes you can do that but you will need a full Jacobian so you will need to do a for loop as you mentioned. Also a small code sample to help you get full Jacobian can be found <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7" rel="nofollow noopener">here</a>.</p>
<p>Note that this gradient penalty will not guarantee that the resulting network will be monotonic non-decreasing. It will just penalize it if it is not.</p>

ElleryL: <p>I know that it doesn’t guarantee since this is just a optimization subject to constraint but I’m fine with it.<br>
Again, the loop isn’t very promising as dimension is so high</p>
<p>As for example for better clarification:  NN( input = [x1,x2,x3,x4,x5,z] ) = [y1,y2,y3,y4,y5,y6,y7]</p>
<p>I want make NN( input = [x1,x2,x3,x4,x5,z+epsilon] ) [i] &gt;= NN( input = [x1,x2,x3,x4,x5,z] ) [i]  for all i and all epsilon &gt; 0</p>

albanD: <p>I’m afraid this is a limitation of using backward mode AD (which all DL framework use) <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
It can only compute vector Jacobian products.</p>

ElleryL: <p>yea; I see; but thanks for the help; I’m gonna think other way of forcing monotonicity than gradient method</p>