The problem of optimization with constraints

username: Squirrel92
cooked: <p>I solve the problem of optimization with constraints. The main problem is that I have two kinds of constraints: the constraints on the solution x and on coeff_matrix.mv(x). It’s a problem-solving code without taking into account constraints. Please, tell me how to modify it to take into account constraints. Here are my constraints:</p>
<p>lower_bounds &lt;= x &lt;= upper_bounds,<br>
lower_constr_bounds &lt; coeff_matrix.mv(x) &lt;= upper_constr_bounds</p>
<p>Here is my code:</p>
<p>import numpy as np</p>
<p>import torch<br>
from torch.optim import LBFGS<br>
from torch.nn import Parameter<br>
import torch.optim as optim</p>
<p>A = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=torch.float64)</p>
<p>def func_cond(x):<br>
x = x.reshape(A.shape[1], A.shape[0])<br>
return torch.linalg.cond(x * A)</p>
<p>n = A.numel()</p>
<p>bounds = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])<br>
bounds.append([…, …])</p>
<p>lower_bounds = torch.tensor([…, …, …, …, …, …, …, …, …], dtype=torch.float64)<br>
upper_bounds = torch.tensor([…, …, …, …, …, …, …, …, …], dtype=torch.float64)</p>
<p>x_start = np.zeros(n)<br>
for i in range(n):<br>
x_start[i] = np.random.uniform(low=bounds[i][0], high=bounds[i][1], size=1)</p>
<p>x = Parameter(torch.tensor(x_start), requires_grad=True)</p>
<p>coeff_matrix = torch.tensor([…, …, …, …, …, …, …, …, …], […, …, …, …, …, …, …, …, …]<br>
[…, …, …, …, …, …, …, …, …], […, …, …, …, …, …, …, …, …],<br>
[…, …, …, …, …, …, …, …, …], […, …, …, …, …, …, …, …, …],<br>
[…, …, …, …, …, …, …, …, …], […, …, …, …, …, …, …, …, …],<br>
[…, …, …, …, …, …, …, …, …], […, …, …, …, …, …, …, …, …], dtype=torch.float64)</p>
<p>lower_constr_bounds = torch.tensor([-np.inf] * 10, dtype=torch.float64)<br>
upper_constr_bounds = torch.tensor([…, …, …, …, …, …, …, …, …, …], dtype=torch.float64)</p>
<p>def closure():<br>
optimizer.zero_grad()<br>
loss = func_cond(x)<br>
loss.backward()<br>
return loss</p>
<p>optimizer = optim.LBFGS(<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>)</p>
<p>for _ in range(100):<br>
optimizer.step(closure)</p>
<p>min_cond = func_cond(x)</p>
<p>print(x)<br>
print(min_cond)</p>

username: tom
cooked: <p>One easy way to approximately account for constraints is to penalize their violation.</p>
<p>If that doesn’t do what you need, active set methods (<a href="https://en.wikipedia.org/wiki/Active-set_method" class="inline-onebox" rel="noopener nofollow ugc">Active-set method - Wikipedia</a>) might be worth a try, but I’m not entirely sure there is a ready-to-use implementation.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Squirrel92
cooked: <p>Thanks a lot! Please could you provide an example of code with penalties for violating constraints. Can it takes into account that I have two kinds of constraints?</p>

username: tom
cooked: <p>If you want c &lt;= 0, you can have <code>tau * c.clamp(min=0)</code> or <code>tau * c.clamp(min=0) ** 2</code> or and add that to your loss. For the other constraints, use translation or unary <code>-</code>.</p>
<p>Best regards</p>
<p>Thomas</p>