How to implement a non decreasing function

hnakao: <p>I would like to learn a piecewise linear non decreasing function where both input and output are one dimensional. This can be achieved by passing through a series of ReLU layers with some constraints on the weights and biases so that the slopes are always non negative.</p>
<p>Can PyTorch implement this? If not, are there any alternative ways?</p>

KFrank: <p>Hi Hnakao!</p>
<aside class="quote no-group" data-username="hnakao" data-post="1" data-topic="150748" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/58f4c7/48.png" class="avatar"> hnakao:</div>
<blockquote>
<p>I would like to learn a piecewise linear non decreasing function</p>
</blockquote>
</aside>
<p>This can certainly be done.</p>
<p>The most natural way to represent a piecewise linear function is as a<br>
linear-interpolation look-up table.  (If each linear piece is non-decreasing,<br>
the overall function will also be non-decreasing.)</p>
<p>One concrete approach is illustrated in the below script.  The basic ideas<br>
are as follows:</p>
<p>Use torch.bucketize to locate <code>input</code> within the look-up table.</p>
<p>Instead of storing the values of the function at the linear-interpolation<br>
“break-point” boundaries, store the log of the differences in the values<br>
of the function from one break-point to the next.  You then get the<br>
difference by exponentiating the log, thereby insuring that the function<br>
is non-decreasing.</p>
<p>(It is possible that it will be easier to train the function differences (slopes)<br>
rather than the logs, but then you would have to impose a positivity<br>
constraint on the differences during training, which can introduce its own<br>
issues.)</p>
<p>In my example, the log-differences (log-slopes) are trainable, but the<br>
break-point boundaries are fixed (not trainable).  They could be made<br>
trainable, but that would introduce significant redundancy, and it’s not<br>
clear that doing so would be beneficial.</p>
<p>Here is an example script that learns an increasing section of the <code>sin()</code><br>
function:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2022)

class TrainableInterpolationTable (torch.nn.Module):   # doesn't need to be a Module
    def __init__ (self, boundaries = None, values = None):
        super().__init__()
        
        if  boundaries is None:
            boundaries = torch.arange (11.)
        if  values is None:
            values = torch.zeros (10)
        
        assert  boundaries.dim() == 1
        assert  values.dim() == 1
        assert  len (values) == len (boundaries) - 1
        
        self.nBin = len (values)   # number of bins in the interpolation table
        self.boundaries = boundaries.clone()   # table-entry boundaries -- must be ascending
        self.values = torch.nn.Parameter (values.clone())   # logs of (positive) linear increments
    
    def forward (self, x):   # apply piecewise linear function element-wise to x
        dt = self.values.dtype
        inds = torch.bucketize (x, self.boundaries)
        inds_cl = torch.clamp (inds, 1, self.nBin)
        expval = self.values.exp()
        y = (inds.unsqueeze (-1) &gt; torch.arange (self.nBin) + 1).to (dtype = dt) @ expval
        y = y + torch.logical_and (inds &gt; 0, inds &lt;= self.nBin).to (dtype = dt) * expval[inds_cl - 1] * (x - self.boundaries[inds_cl - 1]) / (self.boundaries[inds_cl] - self.boundaries[inds_cl - 1])
        return y

tbl_ident = TrainableInterpolationTable()

x = torch.arange (-1.7, 12.0, 2.1)
print ('x = ...')
print (x)
print ('tbl_ident (x) = ...')
print (tbl_ident (x))

bnd = torch.arange (0.0, 1.005, 0.01)
val = (bnd[1:]**2 - bnd[:-1]**2).log()
tbl_square = TrainableInterpolationTable (bnd, val)

x = torch.arange (-0.1, 1.15, 0.101)
print ('x = ...')
print (x)
print ('tbl_square (x) = ...')
print (tbl_square (x))
print ('tbl_square (x) - x**2 = ...')
print (tbl_square (x) - x**2)

x = torch.rand (1000)
sin_x = x.sin()
tbl_train = TrainableInterpolationTable (torch.arange (0.0, 1.005, 0.01), -3 * torch.ones (100))
opt = torch.optim.SGD (tbl_train.parameters(), lr = 0.2, momentum = 0.95)
print ('train tbl_train to learn sine function...')
for  i in range (10001):
    opt.zero_grad()
    mse_loss = torch.nn.functional.mse_loss (tbl_train (x), sin_x)
    mse_loss.backward()
    opt.step()
    if  not i%1000:
        print ('i:', i, '  mse_loss:', mse_loss.item())
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.10.2
x = ...
tensor([-1.7000,  0.4000,  2.5000,  4.6000,  6.7000,  8.8000, 10.9000])
tbl_ident (x) = ...
tensor([ 0.0000,  0.4000,  2.5000,  4.6000,  6.7000,  8.8000, 10.0000],
       grad_fn=&lt;AddBackward0&gt;)
x = ...
tensor([-1.0000e-01,  1.0000e-03,  1.0200e-01,  2.0300e-01,  3.0400e-01,
         4.0500e-01,  5.0600e-01,  6.0700e-01,  7.0800e-01,  8.0900e-01,
         9.1000e-01,  1.0110e+00,  1.1120e+00])
tbl_square (x) = ...
tensor([0.0000e+00, 1.0000e-05, 1.0420e-02, 4.1230e-02, 9.2440e-02, 1.6405e-01,
        2.5606e-01, 3.6847e-01, 5.0128e-01, 6.5449e-01, 8.2810e-01, 1.0000e+00,
        1.0000e+00], grad_fn=&lt;AddBackward0&gt;)
tbl_square (x) - x**2 = ...
tensor([-1.0000e-02,  9.0000e-06,  1.6001e-05,  2.1003e-05,  2.3991e-05,
         2.4989e-05,  2.3991e-05,  2.0981e-05,  1.5974e-05,  9.0003e-06,
         5.9605e-08, -2.2121e-02, -2.3654e-01], grad_fn=&lt;SubBackward0&gt;)
train tbl_train to learn sine function...
i: 0   mse_loss: 5.558672904968262
i: 1000   mse_loss: 0.0014287722297012806
i: 2000   mse_loss: 8.150507346726954e-05
i: 3000   mse_loss: 4.557810825644992e-05
i: 4000   mse_loss: 3.0471503123408183e-05
i: 5000   mse_loss: 2.2024350982974283e-05
i: 6000   mse_loss: 1.6821579265524633e-05
i: 7000   mse_loss: 1.3377467439568136e-05
i: 8000   mse_loss: 1.096794949262403e-05
i: 9000   mse_loss: 9.207196853822097e-06
i: 10000   mse_loss: 7.875628398323897e-06
</code></pre>
<p>Best.</p>
<p>K. Frank</p>