Applying a constraint on tensors during optimization

username: 28utkarsh
cooked: <p>I am optimizing three tensors, a, b, and c, of shape (64, 64, 1) each.</p>
<p>During the forward pass, I want their values such that the following condition holds:</p>
<p>0&lt;= a + b + c &lt;= 1</p>
<p>I don’t know how to update a, b, and c such that the above condition holds true.</p>
<p>Thanks in advance for any help and guidance.</p>

username: KFrank
cooked: <p>Hi Utkarsh!</p>
<aside class="quote no-group" data-username="28utkarsh" data-post="1" data-topic="190875" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/28utkarsh/48/60755_2.png" class="avatar"> 28utkarsh:</div>
<blockquote>
<p>During the forward pass, I want their values such that the following condition holds:</p>
<p>0&lt;= a + b + c &lt;= 1</p>
</blockquote>
</aside>
<p>Using brute force to constrain a trainable parameter tends not to work<br>
nicely with gradient-descent optimization.  I generally find it preferable<br>
to smoothly map unconstrained parameters to dependent variables that<br>
satisfy the desired constraints.</p>
<p><code>sigmoid()</code> smoothly maps the unconstrained range <code>(-inf, inf)</code> to the<br>
constrained range <code>(0.0, 1.0)</code>.</p>
<p>How best to use these ideas to constrain your <code>a + b + c</code> will depend on<br>
your specific use case and what <code>a</code>, <code>b</code>, and <code>c</code> are supposed to mean.</p>
<p>Let me offer a rather elaborate way to achieve this, based on some tacit<br>
assumptions on how you want <code>a</code>, <code>b</code>, and <code>c</code> to behave.  The basic goals of<br>
this particular scheme are to impose the joint constraint that <code>a + b + c</code><br>
lies in the range <code>(0.0, 1.0)</code>, while otherwise leaving <code>a</code>, <code>b</code>, and <code>c</code> as<br>
unconstrained “as possible.”  Also, we want to treat <code>a</code>, <code>b</code>, and <code>c</code> on equal<br>
footing, rather than, say, leaving <code>a</code> and <code>b</code> fully unconstrained and then<br>
constraining <code>c</code> such that the joint constraint is satisfied.  These may or<br>
may not be goals that are appropriate for your use case.</p>
<p>The idea is to start with three trainable parameters that are fully unconstrained<br>
so that they work well with gradient-descent optimization and derive from<br>
them three variables that satisfy the joint constraint that will then be used<br>
for the rest of the forward pass.</p>
<p>To do so, we combine the three unconstrained parameters into a vector<br>
(more precisely, <code>stack()</code> them into a dimension of a multidimensional<br>
tensor), rotate that vector so that the sum of the parameters becomes a<br>
single component of the rotated vector, map that sum to <code>(0.0, 1.0)</code>,<br>
and then rotate back.</p>
<p>This smoothly and differentiably maps the unconstrained parameters to<br>
their constrained versions and treats the three unconstrained parameters<br>
all on the same footing.</p>
<p>Here is a script that implements this scheme:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2023)

a0 = torch.randn (64, 64, 1)                            # unconstrained version of a to train
b0 = torch.randn (64, 64, 1)                            # unconstrained version of b to train
c0 = torch.randn (64, 64, 1)                            # unconstrained version of c to train

print ('(a0 + b0 + c0).min():', (a0 + b0 + c0).min())   # a0, b0, and c0 violate joint constraint
print ('(a0 + b0 + c0).max():', (a0 + b0 + c0).max())   # a0, b0, and c0 violate joint constraint

m = torch.tensor ([
    [ 1.0,  1.0,  1.0],                                 # will form  a + b + c
    [ 0.0,  1.0, -1.0],                                 # chosen to be orthogonal
    [-2.0,  1.0,  1.0]                                  # chosen to be orthogonal
])

m /= m.norm (dim = 1).unsqueeze (-1)                    # make m orthonormal

print ('m @ m.T = ...')                                 # check that m is orthonormal
print (m @ m.T)

abc0 = torch.stack ((a0, b0, c0), dim = -1)

abc = abc0 @ m.T                                        # rotate to "a + b + c" space
abcsum = abc[:, :, :, 0]                                # view of "a + b + c" term
abcmap = (3**0.5 * abcsum).sigmoid() / (3**0.5)         # map  a + b + c  to range (0.0, 1.0)
abcsum.copy_ (abcmap)                                   # copy mapped value back into "a + b + c" view
abc = abc @ m                                           # rotate back to "abc" space

a, b, c = abc.unbind (dim = -1)                         # constrained a, b, and c tensors for forward pass

print ('(a + b + c).min():', (a + b + c).min())         # check that a, b, and c satisfy joint constraint
print ('(a + b + c).max():', (a + b + c).max())         # check that a, b, and c satisfy joint constraint

print ('(a + b).min():', (a + b).min())                 # a, b, and c are "otherwise" unconstrained
print ('(a + b).max():', (a + b).max())                 # a, b, and c are "otherwise" unconstrained
print ('(b + c).min():', (b + c).min())                 # a, b, and c are "otherwise" unconstrained
print ('(b + c).max():', (b + c).max())                 # a, b, and c are "otherwise" unconstrained
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">2.1.0
(a0 + b0 + c0).min(): tensor(-6.3463)
(a0 + b0 + c0).max(): tensor(6.2076)
m @ m.T = ...
tensor([[1.0000, 0.0000, 0.0000],
        [0.0000, 1.0000, 0.0000],
        [0.0000, 0.0000, 1.0000]])
(a + b + c).min(): tensor(0.0018)
(a + b + c).max(): tensor(0.9980)
(a + b).min(): tensor(-2.4540)
(a + b).max(): tensor(3.1374)
(b + c).min(): tensor(-3.1173)
(b + c).max(): tensor(3.1664)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: JosueCom
cooked: <p>If you can turn your constraint into an equality constraint, you can use <a href="https://en.wikipedia.org/wiki/Lagrange_multiplier" class="inline-onebox" rel="noopener nofollow ugc">Lagrange multiplier - Wikipedia</a>. To turn an inequality to an equality constraint, one strategy is to use <a href="https://en.wikipedia.org/wiki/Slack_variable" class="inline-onebox" rel="noopener nofollow ugc">Slack variable - Wikipedia</a>.  For example, you can check how slack variables are used in the simplex method: <a href="https://www.matem.unam.mx/~omar/math340/simplex-intro.html" class="inline-onebox" rel="noopener nofollow ugc">Introduction to the simplex method</a>.</p>