Linear constraints on trainable network parameters

Mathieu: <p>I am defining additional trainable parameters (besides the weights and biases) as <code>nn.ParameterDict(...)</code>. Lets denote these additional parameters as <code>X</code>.</p>
<p>During training, the linear inequality constraint <code>A*X &gt; 0</code> must be satisfied at every epoch. Otherwise, I may not be able to calculate a loss.</p>
<p>Are there built-in solution approaches to handle this? I am using the Adam optimizer, but could also switch to a different one.<br>
(Note: I would like to avoid introducing a loss term for the constraint vialation since I already have some loss terms which I have to weigh appropriately)</p>

KFrank: <p>Hi Mathieu!</p>
<aside class="quote no-group" data-username="Mathieu" data-post="1" data-topic="216334" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathieu/48/74743_2.png" class="avatar"> Mathieu:</div>
<blockquote>
<p>During training, the linear inequality constraint <code>A*X &gt; 0</code> must be satisfied at every epoch. Otherwise, I may not be able to calculate a loss.</p>
</blockquote>
</aside>
<p>Do you really mean “epoch” here?  If you really need your constraint to be satisfied<br>
in order to compute (and then presumably to backpropagate) your loss, I would think<br>
that you would need your constraint to be satisfied for each forward / backward pass<br>
which would typically mean for every batch (not only for an entire epoch).</p>
<aside class="quote no-group">
<blockquote>
<p>Are there built-in solution approaches to handle this?</p>
</blockquote>
</aside>
<p>I am not aware of anything built in for your use case.</p>
<p>I would suggest “projecting” <code>X</code> back to where it satisfies the constraints after<br>
every optimizer step.</p>
<aside class="quote no-group">
<blockquote>
<p><code>A*X &gt; 0</code></p>
</blockquote>
</aside>
<p>I assume that this means <code>A @ X &gt; 0</code> (that is, matrix multiplication) and that <code>A</code> has<br>
shape <code>[m, n]</code> with <code>m &lt; n</code> (that is, there are fewer individual constraints than there<br>
are elements of <code>X</code>).</p>
<p>The constraint violation is <code>V = (A @ X).min (torch.zeros (1))</code>.  Then:</p>
<pre data-code-wrap="python"><code class="lang-python">X_new = X - torch.linalg.lstsq (A, V.unsqueeze (-1)).solution.squeeze()
</code></pre>
<p>will be the closest <code>X_new</code> to <code>X</code> that satisfies <code>A @ X_new &gt;= 0</code>.</p>
<p>Note that if you require that <code>A @ X</code> be strictly greater than zero, not just greater<br>
than or equal to zero  (up to some round-off error), you will have to replace your<br>
constraint with <code>A @ X &gt; eps</code>, for some small, positive <code>eps</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>(Note: I would like to avoid introducing a loss term for the constraint vialation since I already have some loss terms which I have to weigh appropriately)</p>
</blockquote>
</aside>
<p>If I understand your use case, adding a loss term to impose the constraint – whether<br>
as a violation penalty or a Lagrange multiplier – won’t work in any event, as the loss<br>
term will drive <code>X</code> towards satisfying the constraint as the optimization proceeds, but<br>
will not cause the constraint to be satisfied step-by-step during the optimization.  So<br>
if you really need the constraint to be satisfied in order to calculate your loss, you must<br>
reimpose the constraint before each forward / backward pass, which is to say, after<br>
each optimization step.</p>
<p>Best.</p>
<p>K. Frank</p>

Mathieu: <p>Thanks for the detailed answer Frank. Let me go through your statements.</p>
<blockquote>
<p>Do you really mean “epoch” here? If you really need your constraint to be satisfied<br>
in order to compute (and then presumably to backpropagate) your loss, I would think<br>
that you would need your constraint to be satisfied for each forward / backward pass<br>
which would typically mean for every batch (not only for an entire epoch).</p>
</blockquote>
<pre><code class="lang-auto">for epoch in range(epochs):
    optimizer.zero_grad()

    # Forward pass with all samples
    output = model(inputs)

    # Compute loss and backpropagate
    loss = Loss_function(output,...)
    loss.backward()
    optimizer.step()
</code></pre>
<p>What I am currently doing is to pass all samples in one batch to the model. Here, <code>inputs</code> has shape <code>[N, dim]</code> where <code>N</code> denotes the number of samples and <code>dim</code> the sample dimension (input dimension of the NN). Both <code>N</code> and <code>dim</code> are small for my problems, so I am passing all samples in one go to the model. As a consequence – correct me if I am wrong – in that scenario “batch” and “epoch” mean the same thing, right?</p>
<blockquote>
<p>If I understand your use case, adding a loss term to impose the constraint – whether<br>
as a violation penalty or a Lagrange multiplier – won’t work in any event, as the loss<br>
term will drive <code>X</code> towards satisfying the constraint as the optimization proceeds, but<br>
will not cause the constraint to be satisfied step-by-step during the optimization. So<br>
if you really need the constraint to be satisfied in order to calculate your loss, you must<br>
reimpose the constraint before each forward / backward pass, which is to say, after<br>
each optimization step.</p>
</blockquote>
<p>Everything what you said before regarding <code>A</code> and its shape is correct. Sorry for my sloppy notation! Also you are mentioning a good point here. The optimizer will satisfy the constraint while converging to the optimum, but this means that the constraints can be violated at intermediate epochs and I may run into trouble. So projecting onto the feasible set is most likely the way to go. But I am not sure how to include the projection in the epoch loop correctly.</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    optimizer.zero_grad()

    # Forward pass with all samples (X denote the add. params s.t A @ x &gt; 0)
    output, X = model(inputs)
    
    # --&gt; projection should happen here?
    X_new = X - torch.linalg.lstsq (A, V.unsqueeze (-1)).solution.squeeze() 

    # Compute loss and backpropagate
    loss = Loss_function(output, X)
    loss.backward()
    optimizer.step()
</code></pre>
<ol>
<li>Should the projection on the feasible set (using <code>torch.linalg.lstsq</code>) happen after the call to the <code>model</code>, like I sketched it?</li>
<li>If so, does this projection cause problems for the optimizer when calculating the gradients during <code>loss.backward()</code>? Specifically, do we have to “inform” the optimizer about the projection step to make sure the gradients are correct?</li>
</ol>
<p>Best,</p>

KFrank: <p>Hi Mathieu!</p>
<aside class="quote no-group quote-modified" data-username="Mathieu" data-post="3" data-topic="216334" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathieu/48/74743_2.png" class="avatar"> Mathieu:</div>
<blockquote>
<p>What I am currently doing is to pass all samples in one batch to the model.<br>
…<br>
in that scenario “batch” and “epoch” mean the same thing, right?</p>
</blockquote>
</aside>
<p>Yes, if you have only one batch per epoch (and more relevant to the larger<br>
discussion, one optimizer step per epoch), then “batch” and “epoch” become<br>
the same.</p>
<aside class="quote no-group">
<blockquote>
<p>But I am not sure how to include the projection in the epoch loop correctly.</p>
</blockquote>
</aside>
<p>You can put the projection in a number of different, but equivalent places.</p>
<p>First, let me assume that <code>X</code> is a trainable parameter (a leaf variable with<br>
<code>requires_grad = True</code>) that is being optimized by <code>optimizer</code>.</p>
<p>So <code>output, X = model (inputs)</code> is highly suspicious.  <code>X</code> should not be<br>
computed (and therefore presumably not be returned) by <code>model</code>.  Think<br>
through your logic here.  If <code>X</code> has to be returned by <code>model</code>, you’re probably<br>
doing something wrong</p>
<p>I would to it like this:</p>
<pre data-code-wrap="python"><code class="lang-python">...
loss = Loss_function (output, X)
loss.backward()
optimizer.step()
with torch.no_grad():
   V = (A @ X).min (torch.zeros (1))
   X.sub_ (torch.linalg.lstsq (A, V.unsqueeze (-1)).solution.squeeze())
</code></pre>
<p>Note, it is important that <code>X</code> is modified in place, and is modified under the protection<br>
of a <code>with torch.no_grad():</code> block.</p>
<p>The logic is as follows:  <code>optimizer.step()</code> modifies <code>X</code>, possibly causing it to violate<br>
the constraint.  Stylistically, I prefer to reimpose the constraint immediately after<br>
modifying <code>X</code> because I think it makes it clearer what’s going on.  But logically, you<br>
can reimpose the constraint any time before <code>X</code> is used again.</p>
<p>Note, if you call <code>.lstsq()</code> many times without <code>A</code> changing (and if this part of the<br>
processing matters in terms of performance), it may be cheaper to pre-compute the<br>
pseudoinverse of <code>A</code> once, <code>A_inv = A.pinverse()</code>, and reuse it:</p>
<pre data-code-wrap="python"><code class="lang-python">   X.sub_ (A_inv @ V)
</code></pre>
<p>(However, <code>lstsq()</code> will be cheaper unless you reuse <code>A_inv</code> multiple times.)</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>If so, does this projection cause problems for the optimizer when calculating the gradients during <code>loss.backward()</code>? Specifically, do we have to “inform” the optimizer about the projection step to make sure the gradients are correct?</li>
</ol>
</blockquote>
</aside>
<p>No, you don’t need to tell the optimizer – you’re overriding the optimizer by brute<br>
force.  If the optimizer moves <code>X</code> out of compliance with the constraint and you<br>
reimpose is, it is likely that on the next optimization step, the optimizer will similarly<br>
move <code>X</code> out of compliance.  But that’s okay – you just reimpose the constraint<br>
again.  In some sense the optimizer and constraint are working at cross purposes,<br>
but this is generally not a problem.</p>
<p>Best.</p>
<p>K. Frank</p>

Mathieu: <blockquote>
<p>So <code>output, X = model (inputs)</code> is highly suspicious. <code>X</code> should not be<br>
computed (and therefore presumably not be returned) by <code>model</code> . Think<br>
through your logic here. If <code>X</code> has to be returned by <code>model</code> , you’re probably<br>
doing something wrong</p>
</blockquote>
<p>You are absolutely correct. <code>X</code> is a trainable parameter which gets updated by the optimizer and not computed from within the model.</p>
<pre><code class="lang-auto">...
loss = Loss_function (output, X)
loss.backward()
optimizer.step()
with torch.no_grad():
   V = (A @ X).min (torch.zeros (1))
   X.sub_ (torch.linalg.lstsq (A, V.unsqueeze (-1)).solution.squeeze())
</code></pre>
<blockquote>
<p>Note, it is important that <code>X</code> is modified in place, and is modified under the protection<br>
of a <code>with torch.no_grad():</code> block.</p>
</blockquote>
<p>It is not clear to me why the projection should happen within a <code>with torch.no_grad():</code> block. Consider the case without projection where our loss function computes <code>f(X)</code>. During backpropagation, the optimizer (e.g., Adam) computes the gradient of the loss w.r.t <code>X</code>, that is, <code>df(X) / dX =: g(X)</code>.<br>
Lets now include the projection on the feasible set where <code>P(X)</code> is my notation for the projection. Hence, our loss function now computes <code>f(P(X))</code> and the analytical gradient reads <code>grad(P) *  g(P(X))</code> where <code>grad(P)</code> is related to the pseudoinverse of <code>A</code>.</p>
<p>My understanding of <code>with torch.no_grad():</code> is that everything what gets calculated in this block is not accounted for in the gradient calculation. In other words, we created a situation computing a loss <code>f(P(X))</code> and a gradient <code>g(P(X))</code> but the gradient is missing the term <code>grad(P)</code>, hence deviates from the analytical gradient.<br>
Is this correct?</p>
<p>Related to that …</p>
<blockquote>
<p>No, you don’t need to tell the optimizer – you’re overriding the optimizer by brute<br>
force. If the optimizer moves <code>X</code> out of compliance with the constraint and you<br>
reimpose is, it is likely that on the next optimization step, the optimizer will similarly<br>
move <code>X</code> out of compliance. But that’s okay – you just reimpose the constraint<br>
again. In some sense the optimizer and constraint are working at cross purposes,<br>
but this is generally not a problem.</p>
</blockquote>
<p>… should the optimizer and the constraints not work together (rater than at cross purposes)? For instance: The optimizer moves <code>X</code> away from the feasible set, then we reimpose the constraint via brute force. So the optimizer thinks it moved <code>X</code> somewhere, but in reality, it has not. So, similar to what I asked above, why not including the term <code>grad(P)</code> in the backpropagation, thereby letting the optimizer know that we modified <code>X</code>?</p>
<p>I am new to pytorch and stochastic optimizers such as Adam. My background is more on gradient-based optimizers (e.g trust-region solvers) providing analytical gradients along with the loss value. But maybe the pytorch optimizers do not rely on analytical gradients.</p>
<p>Best,</p>

KFrank: <p>Hi Mathieu!</p>
<aside class="quote no-group" data-username="Mathieu" data-post="5" data-topic="216334" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathieu/48/74743_2.png" class="avatar"> Mathieu:</div>
<blockquote>
<p>It is not clear to me why the projection should happen within a <code>with torch.no_grad():</code> block.</p>
</blockquote>
</aside>
<p>Any modification of the trainable parameter <code>X</code> has to occur under <code>.no_grad()</code><br>
protection.  So if you set <code>X</code> itself to its projected value, you need the block.</p>
<aside class="quote no-group">
<blockquote>
<p>Lets now include the projection on the feasible set where <code>P(X)</code> is my notation for the projection. Hence, our loss function now computes <code>f(P(X))</code></p>
</blockquote>
</aside>
<p>Yes, you could do this instead.  Writing out some of the steps:</p>
<pre data-code-wrap="python"><code class="lang-python">X_proj = P (X)
loss = Loss_function (output, X_proj)
loss.backward()
</code></pre>
<p>(Note, you could implement <code>X_proj = P (X)</code> as a <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html" rel="noopener nofollow ugc">parametrization.</a>  This is really<br>
just a convenience layer that doesn’t differ substantively from imposing the constraint<br>
“by hand.”)</p>
<aside class="quote no-group">
<blockquote>
<p>My understanding of <code>with torch.no_grad():</code> is that everything what gets calculated in this block is not accounted for in the gradient calculation.</p>
</blockquote>
</aside>
<p>This is correct.</p>
<aside class="quote no-group">
<blockquote>
<p>why not including the term <code>grad(P)</code> in the backpropagation, thereby letting the optimizer know that we modified <code>X</code>?</p>
</blockquote>
</aside>
<p>There are use cases where not including the <code>grad (P)</code> term empirically works.<br>
There are also use cases where including <code>grad (P)</code> empirically works.  I am not<br>
aware of any head-to-head comparisons showing that one approach in generally<br>
better than the other,</p>
<p>You could try both on your problem and see whether you get better training with one<br>
or the other.</p>
<p>(Note that when you impose a constraint, rather than add a constraint penalty to<br>
your loss function, you open up the possibility that certain “modes” of the unconstrained<br>
parameter could start drifting off to infinity, potentially leading to excessive round-off<br>
error or overflow.  In such a case it would be prudent to add some regularization such<br>
as weight decay to your training.  Mild weight decay is generally innocuous and is<br>
often helpful for other reasons.)</p>
<aside class="quote no-group">
<blockquote>
<p>I am new to pytorch and stochastic optimizers such as Adam.<br>
…<br>
But maybe the pytorch optimizers do not rely on analytical gradients.</p>
</blockquote>
</aside>
<p>Just a semantic quibble:  Pytorch optimizers “rely” on mathematically correct, up to<br>
round-off error, gradients computed numerically by autograd, but not technically on<br>
analytical gradients.</p>
<p>But, depending on the details of the use case, those gradients don’t always need to<br>
be the actual (numerically correct) gradients.  They can be approximate or sometimes<br>
willfully modified.  For example, the gradients computed for one batch will differ from<br>
the gradients computed from the next batch in the training loop.  Depending on your<br>
point of view, you can consider this to be a tolerable approximation or to be a useful<br>
source of stochasticity that improves training and the performance of your final model.</p>
<p>As an aside, <code>Adam</code> is a good optimizer that I often use.  But when starting with a<br>
new problem, or experimenting with things like including <code>grad (P)</code> or not, I generally<br>
start with plain-vanilla <code>SGD</code> (with no momentum), tune the learning rate a little, and<br>
try adding things like momentum and weight decay.  Only after I get a general sense<br>
of how training progresses, might I switch to <code>Adam</code>.  At a minimum, <code>SGD</code> is easier to<br>
reason about.</p>
<p>Best.</p>
<p>K. Frank</p>

Mathieu: <p>Thanks for your detailed and valuable answer again! Just two small questions:</p>
<blockquote>
<p>There are use cases where not including the <code>grad (P)</code> term empirically works.<br>
There are also use cases where including <code>grad (P)</code> empirically works. I am not<br>
aware of any head-to-head comparisons showing that one approach in generally<br>
better than the other,</p>
<p>You could try both on your problem and see whether you get better training with one<br>
or the other.</p>
</blockquote>
<p>If I understood correctly, then there is no chance to include <code>grad(P)</code> as I have to write the projection in a <code>with torch.no_grad():</code>. Hence, the projection will not be tracked and not be accounted for in the grad calculation.</p>
<blockquote>
<p>(Note that when you impose a constraint, rather than add a constraint penalty to<br>
your loss function, you open up the possibility that certain “modes” of the unconstrained<br>
parameter could start drifting off to infinity, potentially leading to excessive round-off<br>
error or overflow. In such a case it would be prudent to add some regularization such<br>
as weight decay to your training. Mild weight decay is generally innocuous and is<br>
often helpful for other reasons.)</p>
</blockquote>
<p>Good point. To prevent this, is it possible to impose bounds on <code>X</code> without in-place modifications? Clearly, brute forcing them (like the projection logic) could be realized after every optimizer step via</p>
<pre><code class="lang-auto">with torch.no_grad():
     X_new[:] = X.clamp(lb,ub)
</code></pre>
<p>There is also the <code>sigmoid</code> trick:</p>
<pre><code class="lang-auto">X = torch.tensor(value)
X = torch.nn.Parameter(lb + torch.sigmoid(X) * (ub - lb),  requires_grad=True)
</code></pre>
<p>But is there a direct way to tell the optimizer: “when optimizing <code>X</code>, make sure that <code>optimizer.step()</code> respects the bounds”. If that were possible, then we would not open up the possibility that certain modes of the unconstrained problem drift to infinity.</p>
<p>Best,</p>

Mathieu: <p><a class="mention" href="/u/kfrank">@KFrank</a> Not sure if you are still following this discussion here. If so, I appreciated your advise on the following:</p>
<p>Based on what you stated, we can implement the projection as</p>
<pre><code class="lang-auto">    with torch.no_grad():
         V = (A @ X).min (torch.zeros (1)) 
         param.sub_ (A.pinverse() @ V)
</code></pre>
<p>This means we are building the pseudo-inverse of the entire <code>A</code> matrix. To project onto the set of active constraints, would it not be required to build the pseudo-inverse of  <code>A_sub</code> only, where <code>A_sub</code> contains only the rows of <code>A</code> associated with active constraints ?</p>

KFrank: <p>Hi Mathieu!</p>
<aside class="quote no-group" data-username="Mathieu" data-post="8" data-topic="216334" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathieu/48/74743_2.png" class="avatar"> Mathieu:</div>
<blockquote>
<pre><code class="lang-auto">    with torch.no_grad():
         V = (A @ X).min (torch.zeros (1)) 
         param.sub_ (A.pinverse() @ V)
</code></pre>
<p>This means we are building the pseudo-inverse of the entire <code>A</code> matrix. To project onto the set of active constraints, would it not be required to build the pseudo-inverse of  <code>A_sub</code> only, where <code>A_sub</code> contains only the rows of <code>A</code> associated with active constraints ?</p>
</blockquote>
</aside>
<p>Yes, this is correct.</p>
<p>That is why I suggested in my earlier post using <code>torch.linalg.lstsq (A, V)</code> (with<br>
appropriate <code>squeeze()</code>s) instead of <code>A.pinverse() @ V</code>.  Very crudely, <code>linalg.lstsq()</code><br>
is sort of like performing <code>pinverse()</code>, but just for the active constraints.</p>
<p>(I don’t think this is your use case, but if you were to call  <code>torch.linalg.lstsq (A, V)</code><br>
for multiple values of <code>V</code>, but for the same value of <code>A</code>, then it could be cheaper to call<br>
<code>A.pinverse()</code> once and reuse that result for the multiple values of <code>V</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

Mathieu: <p><a class="mention" href="/u/kfrank">@KFrank</a>  Thank you!</p>
<aside class="quote no-group" data-username="KFrank" data-post="9" data-topic="216334">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(I don’t think this is your use case, but if you were to call <code>torch.linalg.lstsq (A, V)</code><br>
for multiple values of <code>V</code>, but for the same value of <code>A</code>, then it could be cheaper to call<br>
<code>A.pinverse()</code> once and reuse that result for the multiple values of <code>V</code>.)</p>
</blockquote>
</aside>
<p>The projection costs are indeed negligible in my case. <code>X</code> has not more than 10-50 elements.</p>
<p>You also mentioned a subtle detail:</p>
<aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="216334">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(Note that when you impose a constraint, rather than add a constraint penalty to<br>
your loss function, you open up the possibility that certain “modes” of the unconstrained<br>
parameter could start drifting off to infinity, potentially leading to excessive round-off<br>
error or overflow. In such a case it would be prudent to add some regularization such<br>
as weight decay to your training. Mild weight decay is generally innocuous and is<br>
often helpful for other reasons.)</p>
</blockquote>
</aside>
<p>What I am observing is that the optimizer moves the parameter far away from the constrained region because it does not know about their existence since we are overriding the parameters by brute force. To address this, I believe it is prudent to add a penalty term</p>
<pre><code class="lang-auto">L_p= ||X - X_proj ||^2 
</code></pre>
<p>where <code>X</code> and <code>X_proj</code> are the parameters before the projection and the projected parameters, respectively. But the calculation of <code>X_proj</code> occurs within <code>with torch.no_grad():</code>, essentially translating into <code>dX_proj / dX = 0</code>. But still, by means of <code>L_p</code>, we are defining a “target” we want the optimizer to stick close to and <code>dX_proj / dX = 0</code> may not be a problem.</p>
<p>What do you think about the argumentation and idea of having the projection loss term?<br>
Can you think of easier approaches (having less loss terms is in general a good idea) to guide the optimizer to move not too far away from the constrained space?</p>

KFrank: <p>Hi Mathieu!</p>
<aside class="quote no-group" data-username="Mathieu" data-post="10" data-topic="216334" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathieu/48/74743_2.png" class="avatar"> Mathieu:</div>
<blockquote>
<p>What I am observing is that the optimizer moves the parameter far away from the constrained region because it does not know about their existence since we are overriding the parameters by brute force. To address this, I believe it is prudent to add a penalty term</p>
</blockquote>
</aside>
<p>Do you have any evidence that having the optimizer move the parameter outside of<br>
the constraint causes any actual problem?  I would typically advise against adding the<br>
complication of a penalty term unless you have evidence that it is necessary.</p>
<p>From what you said above, you need in any event the brute-force projection – the penalty<br>
term in isolation won’t be enough.  Unless you can show that just using the projection<br>
doesn’t work, why try to fix something that isn’t broken?</p>
<p>As an aside (along the lines of don’t let your “fix” break something else), let’s say that<br>
prior to projection, your <code>X</code> satisfies the constraint, that is, <code>X = X_proj</code>.  Let’s also say<br>
the optimizer step, without the penalty, would give some <code>X_new</code> as the updated <code>X</code>.  And<br>
just to make the discussion simpler, let’s say that <code>X_new</code> itself satisfies the constraint<br>
(without projection).  If you turn on the penalty, the penalty will cause <code>X_new</code>to be closer<br>
to <code>X</code> than it “should” have been, giving you a sub-optimal update.  Would the presumed<br>
benefit of the penalty outweigh this disadvantage?</p>
<p>Best.</p>
<p>K. Frank</p>