Is there a better way to generate monotone positive trainable weights?

Ximeng: <p>Hi , I was trying to generate a group of trainable monotone positive tensors as the weights for Conv1d. It actually works to some extent, but the problem is the real trainable parameter “alpha”/“beta” changed very slightly compared with orthodox convolution weights. They just fluctuated around the initiate value. So I suspect the grads are not very friendly to train due to the way I create those weights. Just curious if there is better way to handle it. Thanks!</p>
<pre><code class="lang-auto">class Fir1d(nn.Module):
    def __init__(self, k_size, device, init_alpha, init_beta=0):
        super().__init__()
        # every channel uses same kernel weights
        assert k_size % 2 == 1
        m = (k_size-1)//2
        self.leftpad = nn.ReplicationPad1d((k_size-1, 0))
        self.lin = torch.linspace(-m, m, k_size, device=device, requires_grad=False)
        self.alpha = nn.Parameter(torch.tensor([init_alpha], device=device, dtype=torch.float32, requires_grad=True))
        self.beta = nn.Parameter(torch.tensor([init_beta], device=device, dtype=torch.float32, requires_grad=True))
        self.w = torch.softmax(self.lin * self.alpha + self.beta, dim=0).unsqueeze(0).unsqueeze(0)  # 1,1,k


    def forward(self, x):
        # b,c,l
        xlist = []
        x = self.leftpad(x)
        B,C,L = x.shape
        for i in range(C):
            subx = x[:, [i], :]
            main_trend = F.conv1d(subx, self.w)
            xlist.append(main_trend)
        xlist = torch.cat(xlist, dim=1)
        return xlist</code></pre>

KFrank: <p>Hi Ximeng!</p>
<aside class="quote no-group" data-username="Ximeng" data-post="1" data-topic="152301" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/8c91f0/48.png" class="avatar"> Ximeng:</div>
<blockquote>
<p>I was trying to generate a group of trainable monotone positive tensors as the weights for Conv1d.</p>
</blockquote>
</aside>
<p>Am I correct that you want the individual weight-values in your <code>conv1d()</code><br>
kernel to be positive and monotonically increasing, even as they train?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">class Fir1d(nn.Module):
    def __init__(self, k_size, device, init_alpha, init_beta=0):
        ...
        self.lin = torch.linspace(-m, m, k_size, device=device, requires_grad=False)
        self.alpha = nn.Parameter(torch.tensor([init_alpha], device=device, dtype=torch.float32, requires_grad=True))
        self.beta = nn.Parameter(torch.tensor([init_beta], device=device, dtype=torch.float32, requires_grad=True))
        self.w = torch.softmax(self.lin * self.alpha + self.beta, dim=0).unsqueeze(0).unsqueeze(0)  # 1,1,k
</code></pre>
</blockquote>
</aside>
<p>This won’t work.</p>
<p>You are only calling:</p>
<pre data-code-wrap="python"><code class="lang-python">self.w = torch.softmax(self.lin * self.alpha + self.beta, dim=0).unsqueeze(0).unsqueeze(0)
</code></pre>
<p>once, when you initialize your <code>Fir1d</code> model.  Even though you update <code>alpha</code><br>
and <code>beta</code> when you run your optimize step, you never recompute <code>w</code> (your<br>
kernel weights), so changing the values of <code>alpha</code> and <code>beta</code> doesn’t actually<br>
do anything.</p>
<p>Recompute <code>w</code> inside of <code>forward()</code> (and just have it be a local variable of<br>
<code>Fir1d</code>’s <code>forward()</code> method), e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">        ...
        w = torch.softmax(self.lin * self.alpha + self.beta, dim=0).unsqueeze(0).unsqueeze(0)
        for i in range(C):
            subx = x[:, [i], :]
            main_trend = F.conv1d (subx, w)
            xlist.append(main_trend)
        xlist = torch.cat(xlist, dim=1)
        return xlist
</code></pre>
<p>or, probably more efficiently, without the loop:</p>
<pre data-code-wrap="python"><code class="lang-python">    def forward(self, x):
        w = torch.softmax (self.lin * self.alpha, dim=0).unsqueeze(0).expand (C, 1, self.lin.size (0))
        return F.conv1d (x, w, groups = C)
</code></pre>
<p>I left <code>self.beta</code> out of the loop-free version because it doesn’t do anything.<br>
<code>softmax()</code> takes “raw-score” logits that it then, in effect, “normalizes,” causing<br>
<code>self.beta</code> to drop out of the <code>softmax()</code> computation.</p>
<p>So (even if you leave <code>beta</code> in) you will be training only one kernel-weight<br>
parameter, <code>alpha</code>.</p>
<p>As an aside, if you wanted your kernel weights to depend on more trainable<br>
parameters, while still being positive and increasing monotonically, you could:</p>
<pre data-code-wrap="python"><code class="lang-python">    def __init__(self, k_size, device, init_alpha, init_beta=0):
        ...
        self.kernel_parameters = nn.Parameter (torch.zeros (k_size))   # initial value

    def forward(self, x):
        w = self.kernel_parameters.exp().cumsum (0).unsqueeze (0).expand (C, 1, kernel_parameters.size (0))
        return F.conv1d (x, w, groups = C)
</code></pre>
<p>Your raw <code>kernel_parameters</code> run from <code>-inf</code> to <code>inf</code>.  <code>.exp()</code> will cause your<br>
derived weights, <code>w</code>, to be positive, and <code>.cumsum()</code> will cause your derived<br>
weights to be monotonically increasing.</p>
<p>Best.</p>
<p>K. Frank</p>

Ximeng: <p>Thank you Frank! You are so professional. Every time you helped me a lot XD.</p>