How to implement Lagrange Multiplier with PyTorch?

username: sakuraiiiii
cooked: <p>Hi, I have a question about how to implement the universal Lagrange Multiplier and KKT conditions with PyTorch. Specificlly, the optimziation problem is<br>
$$<br>
\min f(x), s.t. h(x)=0, g(x)&lt;=0<br>
$$<br>
Although I know the method to solve this problem but I don’t know how to implement it with PyTorch. Could you give me an example? Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>It depends which methods you want to use to solve this?<br>
In general, there are only unconstrained, gradient descent based optimizers in pytorch as it is what can be done efficiently for neural nets.</p>

username: sakuraiiiii
cooked: <p>I just want to do an optimization problem with constriants. With Lagrange Multiplier and KKT conditions, the problem could be converted to an unconstrained problem. I think this problem could be optimized with PyTorch. But there are many conditions for the lagrange multiplier, so I don’t know how to implement it. Could you give me an example? Thank you.</p>

username: albanD
cooked: <p>I don’t have an example implementation from the top of my mind.<br>
If you have a specific algorithm in mind you might be able to find an implementation online though.</p>
<p>Whatever algorithm you want to use to solve your unconstrained problem, you can use pytorch to get gradients and/or perform the steps you need.</p>
<blockquote>
<p>But there are many conditions for the lagrange multiplier, so I don’t know how to implement it.</p>
</blockquote>
<p>Lagrage Multipliers is just one way to rewrite the problem. You still need to select an optimization algorithm to solve this new problem.</p>

username: KFrank
cooked: <p>Hiiiii Sakuraiiiii!</p>
<aside class="quote no-group" data-username="sakuraiiiii" data-post="3" data-topic="89011" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sakuraiiiii/48/19481_2.png" class="avatar"> sakuraiiiii:</div>
<blockquote>
<p>I just want to do an optimization problem with constriants. With Lagrange Multiplier and KKT conditions, the problem could be converted to an unconstrained problem.</p>
</blockquote>
</aside>
<p>To expand a little on what Alban has said, the problem is as follows:</p>
<p>If I have an optimization problem where I seek to <em>minimize</em> a loss<br>
function (e.g., a neural-network training loss), but then add constraints<br>
to it, it is true that I can use Lagrange multipliers to convert the<br>
constrained optimization to an unconstrained optimization.</p>
<p>However, this unconstrained optimization problem is no longer a<br>
<em>minimization</em> problem, but rather one of finding a <em>stationary point</em><br>
(also called a <em>critical point</em>) of the augmented loss function.</p>
<p>That is, the gradient (including the gradient with respect to the<br>
Lagrange multipliers) will vanish at the stationary-point solution, but<br>
this solution need not be a minimum (with respect to the Lagrange<br>
multipliers).</p>
<p>A simple example illustrates the difference:</p>
<p>We can minimize <code>x**2</code> with gradient descent just fine, and the minimum<br>
occurs ar <code>x = 0</code> where the derivative vanishes.</p>
<p>The function <code>x**3</code>  also has a stationary point at <code>x = 0</code> (where the<br>
derivative vanishes), but gradient descent is no good for finding it.<br>
Once <code>x</code> ends up less than zero – even by a little bit – gradient descent<br>
will keep pushing <code>x</code> to become more and more negative, and off to<br>
<code>-inf</code>.</p>
<p>I don’t know enough to state that you <em>can’t</em> use gradient descent with<br>
Lagrange multipliers, but I don’t know how to do it.</p>
<p>If I were to try to hack something together for your problem, I would<br>
try adding penalties for constraint violation to your loss function, and<br>
use pytorch / gradient descent to minimize that.</p>
<p>So you could add, for example, <code>alpha * h(x)**2</code> for your equality<br>
constraints, and <code>beta * min (g(x), 0)**2</code> for your inequality<br>
constraints.  As you increase <code>alpha</code> and <code>beta</code>, minimizing your loss<br>
function will increasing push you in the direction of satisfying your<br>
constraints.  (Perhaps <code>abs()</code> would be better than <code>**2</code>, as it avoids<br>
the “softer” quadratic minimum.)</p>
<p>You could say that adding these penalties to you loss function will<br>
“encourage” your constraints to be close to being satisfied, but it won’t<br>
force them to be satisfied.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: sakuraiiiii
cooked: <p>Hi, KFrank. Thank you very much. I will follow you advice. But according to what you said, I have another question that maybe I can minimize abs(\nambla L) whose minimum is 0, where L=f(x)+\lambda h(x) + \mu g(x). Is it will be worked?</p>

username: KFrank
cooked: <p>Hiiiii Sakuraiiiii!</p>
<aside class="quote no-group" data-username="sakuraiiiii" data-post="6" data-topic="89011" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sakuraiiiii/48/19481_2.png" class="avatar"> sakuraiiiii:</div>
<blockquote>
<p>I can minimize abs(\nambla L) whose minimum is 0, where L=f(x)+\lambda h(x) + \mu g(x). Is it will be worked?</p>
</blockquote>
</aside>
<p>In principle, you could do this.  The problem (in general, and in<br>
the specific context of pytorch) is how are you going to calculate<br>
<code>abs (nabla L)</code>?</p>
<p>If you can do it analytically, great.  (But then you might even be able<br>
to solve your full optimization problem analytically.)</p>
<p>You could imagine using pytorch’s autograd to numerically calculate<br>
<code>nabla L</code>, but if you then want to use gradient descent to minimize<br>
<code>abs (nabla L)</code>, you will be, in effect, calculating second derivatives<br>
of <code>L</code> with respect to the parameters of your model and your Lagrange<br>
multiplier.</p>
<p>Research using and calculating the <em>Hessian matrix</em> in pytorch to<br>
see what people have been able to do with this and the difficulties<br>
involved.</p>
<p>Best.</p>
<p>K. Frank</p>

username: sakuraiiiii
cooked: <p>You’re right. Thank you very much.</p>

username: tom
cooked: <aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="89011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>That is, the gradient (including the gradient with respect to the<br>
Lagrange multipliers) will vanish at the stationary-point solution, but<br>
this solution need not be a minimum (with respect to the Lagrange<br>
multipliers).</p>
</blockquote>
</aside>
<p>Note that here, we know w.r.t. which variables the Lagrangian would be maximized and w.r.t. which ones it would be minimized, so in this way the problem is much easier than “find a saddle point”.</p>
<p>Indeed, while there are few guarantees (though some papers seem to appear, in particular for convex/concave problems) an alternating optimization scheme where you take descent steps for the minimization vars and ascent steps in the maximization vars could be expected to find something (this is basically what the GAN does too in Gparam = argmin_Gparam max_Dparam E(D(x_true, x_fake)), and from there we know that balancing the two types of steps is important, more so for us because the maximization is unconstrained by construction - as it’s designed to take max = infinity for violation of the linear constraints).</p>
<p>For the inequality (KKT) conditions, you could probably use an <a href="https://en.wikipedia.org/wiki/Active-set_method" rel="noopener nofollow ugc">active set method</a>.</p>
<p>Fun fact, and here we go back to <a class="mention" href="/u/kfrank">@KFrank</a>’s advice to use penalty terms: If memory serves me well, the active-set-method of Lawson and Hanson for <a href="https://en.wikipedia.org/wiki/Non-negative_least_squares" rel="noopener nofollow ugc">NNLS</a> with linear constraints uses a 1/epsilon (“machine precision”) penalty for the equality constraints.</p>
<p>Best regards</p>
<p>Thomas</p>

username: KFrank
cooked: <p>Hi Thomas!</p>
<aside class="quote no-group" data-username="tom" data-post="9" data-topic="89011" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>Note that here, we know w.r.t. which variables the Lagrangian would be maximized and w.r.t. which ones it would be minimized, so in this way the problem is much easier than “find a saddle point”.</p>
</blockquote>
</aside>
<p>I don’t think it is as simple as this, the reason being that at the<br>
constrained minimum, the gradient of the loss function plus the<br>
Lagrange-multiplier terms in the direction of the Lagrange<br>
multiplier is flat.</p>
<p>Consider the trivial example:</p>
<p>Minimize <code>x**2 / 2</code>, where <code>x = 1</code>.  To do so, we find the critical<br>
point of <code>x**2 / 2 + lambda * (x - 1)</code>.  The critical point is<br>
<code>(1, -1)</code>, and at the critical point the Hessian matrix is<br>
<code>[[1, 1], [1, 0]]</code>.  (The second partial with respect to <code>lambda</code><br>
is indeed 0.)</p>
<p>Letting <code>phi = (1 + sqrt (5)) / 2</code> (the so-called “golden<br>
ratio”), the Hessian’s eigenvectors (“principal axes”) are<br>
<code>(phi, 1)</code> and  <code>(1, -phi)</code> (positive and negative curvature,<br>
receptively).  These don’t align with the original variable, <code>x</code>, and<br>
the Lagrange multiplier.</p>
<p>Even near the constrained minimum, I don’t see how you will<br>
know in which directions to take “descent” steps and “ascent”<br>
steps without calculating the second partials and looking at<br>
the Hessian matrix.</p>
<p>Best.</p>
<p>K. Frank</p>

username: tom
cooked: <p>I could be wrong, but I think that while yes, you won’t get optimal directions for the steps (which isn’t surprising when using SGD), it would still find the right directions <em>because</em> the structure of the problem min_x max_lambda L says we can minimize w.r.t. x and maximize w.r.t. lambda.</p>
<pre data-code-wrap="python"><code class="lang-python">lfn = lambda x, l: x**2 / 2 + l * (x - 1)
x = torch.tensor(-0.1, requires_grad=True)
l = torch.tensor(0.0, requires_grad=True)

xs = []
ls = []
for i in range(200):
    f = lfn(x, l)
    dx, dl = torch.autograd.grad(f, [x, l])
    with torch.no_grad():
        x -= 1e-1 * dx
        l += 1e-1 * dl
    xs.append(x.item())
    ls.append(l.item())

%matplotlib inline
from matplotlib import pyplot
pyplot.plot(xs)
pyplot.plot(ls)
x.item()
</code></pre>
<p>gives <code>0.9999565482139587</code>.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd05f0983afb9a95bb8137cf61369c32c7dfa65a.png" alt="image" data-base62-sha1="tfIBF3fNKqj6nxl1B7Lg1vRO70u" width="380" height="248"></p>
<p>Best regards</p>
<p>Thomas</p>

username: KFrank
cooked: <p>Hi Thomas!</p>
<aside class="quote no-group" data-username="tom" data-post="11" data-topic="89011" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>it would still find the right directions <em>because</em> the structure of the problem min_x max_lambda L says we can minimize w.r.t. x and maximize w.r.t. lambda.</p>
</blockquote>
</aside>
<p>Thanks, you’re right.  I stand corrected.</p>
<p>See, for example, “Constrained Differential Optimization,” Platt<br>
and Barr, 1988:</p>
<p><a href="https://papers.nips.cc/paper/4-constrained-differential-optimization.pdf" rel="noopener nofollow ugc">https://papers.nips.cc/paper/4-constrained-differential-optimization.pdf</a></p>
<p>Best.</p>
<p>K. Frank</p>