How to do constrained optimization in PyTorch

TinfoilHat0: <p>What’s the proper way to do constrained optimization in PyTorch?</p>
<p>For example, I want each parameter of my model to be bounded both from above and below by some constants cLow and cHigh.</p>
<p>That is, if W is the d-dimensional (flattened) weight vector of my model, I’d like to enforce<br>
cLow &lt; W[i] &lt; cHigh for i = 1, 2, … d. How can I do that?</p>

crowsonkb: <p>You can do projected gradient descent by enforcing your constraint after each optimizer step. An example training loop would be:</p>
<pre><code class="lang-python">    opt = optim.SGD(model.parameters(), lr=0.1)
    for i in range(1000):
        out = model(inputs)
        loss = loss_fn(out, labels)
        print(i, loss.item())
        opt.zero_grad()
        loss.backward()
        opt.step()
        with torch.no_grad():
            for param in model.parameters():
                param.clamp_(-1, 1)
</code></pre>
<p>The last three lines enforce the constraint that the weights fall in the range -1–1.</p>

VIVEK_RUHELA: <p>I am also working on constraints optimization problems. My experience with this suggestion is not positive. When I don’t use <code>clamp_()</code> and train model with no restriction then value of specific weights of interested is close to desired and model is predicting good results. But after using <code>clamp_()</code>, model performance degrade severely. What can be the possible reason for this.</p>

oricha: <p>One reason is that this clamping is not communicated to the optimizer, and in particular destroys the gradients. So the optimizer falsely believes that it has moved the parameter in a certain direction, when in fact it is clamped to the same value as before.</p>

Max_Unhold: <p><a class="mention" href="/u/oricha">@oricha</a> Did you find any solution to that problem?</p>

Grant_Norman: <p>I’ve worked quite a bit on combining PyTorch with constrained optimization. On one hand, my constraints are nonlinear functions of the parameters, as opposed to bound constraints here. The temporary solution I decided upon was using SciPy’s optimize routines, and using PyTorch to compute gradients.</p>
<p>The code for this was part of a more general <a href="https://arxiv.org/abs/2406.02581" rel="noopener nofollow ugc">paper</a>, but the main function is <a href="https://github.com/gnorman7/constrained_nn_eq_discovery/blob/main/src/constrained_nn_eq_discovery/con_opt.py" rel="noopener nofollow ugc">here</a>.</p>
<p>Another option would be a similar (more professional) wrapper of <a href="https://petsc.org/release/overview/tao_solve_table/#bound-constrained" rel="noopener nofollow ugc">PETSc</a>:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://gitlab.com/stefanozampini/petscml/-/tree/main?ref_type=heads">
  <header class="source">
      <img src="https://gitlab.com/assets/favicon-72a2cad5025aa931d6ea56c3201d1f18e68a8cd39788c7c80d5b2b82aa5143ef.png" class="site-icon" width="32" height="32">

      <a href="https://gitlab.com/stefanozampini/petscml/-/tree/main?ref_type=heads" target="_blank" rel="noopener nofollow ugc">GitLab</a>
  </header>

  <article class="onebox-body">
    <img width="64" height="64" src="https://gitlab.com/assets/twitter_card-570ddb06edf56a2312253c5872489847a0f385112ddbcd71ccfa1570febab5d2.jpg" class="thumbnail onebox-avatar">

<h3><a href="https://gitlab.com/stefanozampini/petscml/-/tree/main?ref_type=heads" target="_blank" rel="noopener nofollow ugc">Files · main · Stefano Zampini / petscml · GitLab</a></h3>

  <p>GitLab.com</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>