How to make the parameter of torch.nn.Threshold learnable?

dlmacedo: <p>I the following code, I would like to make <em>delta</em> a parameter learnable by the model instead of a fixed scalar value.</p>
<p>I would like to have the option of a learnable <em>delta</em> for each component of the input tensor or for each layer.</p>
<blockquote>
<pre><code> self.features = nn.Sequential(
      nn.Conv2d(3, 64, kernel_size=5, stride=1, padding=2),
      nn.Threshold(-delta, -delta, inplace=True),
      nn.MaxPool2d(kernel_size=3, stride=2),
      nn.Conv2d(64, 192, kernel_size=5, padding=2),
      nn.Threshold(-delta, -delta, inplace=True),
      nn.MaxPool2d(kernel_size=3, stride=2),
 )
</code></pre>
</blockquote>

hughperkins: <p>I tried using torch.clamp, but also seems non-differentiable:</p>
<pre><code>import torch
from torch import autograd

threshold = autograd.Variable(torch.rand(1), requires_grad=True)
print('threshold', threshold)
# m = torch.nn.Threshold(threshold, threshold)
input = autograd.Variable(torch.rand(1, 5), requires_grad=True) - 0.5
print('input', input)
# out = m(input)
out = torch.clamp(input, min=threshold)
print('out', out)
out.backward(torch.ones(1, 5))
print('threshold.grad.data', threshold.grad.data)

&gt; Traceback (most recent call last):
&gt;   File "4729.py", line 11, in &lt;module&gt;
&gt;     out = torch.clamp(input, min=threshold)
&gt;   File "/Users/hugh2/conda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 396, in clamp
&gt;     return CmaxConstant(min)(self)
&gt;   File "/Users/hugh2/conda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/pointwise.py", line 232, in forward
&gt;     self._max_buffer = i.gt(self.constant).type_as(i)
&gt; TypeError: gt received an invalid combination of arguments - got (Variable), but expected one of:
&gt;  * (float value)
&gt;       didn't match because some of the arguments have invalid types: (Variable)
&gt;  * (torch.FloatTensor other)
&gt;       didn't match because some of the arguments have invalid types: (Variable)
</code></pre>
<p>I tried on tensorflow, and seemed to work ok:</p>
<pre><code>import tensorflow as tf

graph = tf.Graph()
with graph.as_default():
    input_t = tf.placeholder(tf.float32, [None], 'input')
    threshold_t = tf.Variable(0.05)
    out_t = tf.minimum(input_t, threshold_t)
    sess = tf.Session()
    with sess.as_default():
        sess.run(tf.global_variables_initializer())
        print('out', sess.run(out_t, feed_dict={input_t: [-0.3, 0.0, 0.7]}))

        # get grad of out_t wrt threshold_t
        grad_out_t = tf.gradients(out_t, [threshold_t])[0]
        print('d(out)/d(theshold)', sess.run(grad_out_t, feed_dict={input_t: [-0.3, 0.0, 0.7]}))
        print('d(out)/d(theshold)', sess.run(grad_out_t, feed_dict={input_t: [-0.3, 0.0, -0.7]}))
        print('d(out)/d(theshold)', sess.run(grad_out_t, feed_dict={input_t: [-0.3, 0.5, 0.7]}))

out [-0.30000001  0.          0.05      ]
d(out)/d(theshold) 1.0
d(out)/d(theshold) 0.0
d(out)/d(theshold) 2.0
</code></pre>
<p>Edit: I guess maybe this needs the Scalar thing that’s on the way, in order to be solved?</p>

tom: <aside class="quote no-group" data-username="hughperkins" data-post="2" data-topic="4729">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hughperkins/48/915_2.png" class="avatar"> hughperkins:</div>
<blockquote>
<p>out = torch.clamp(input, min=threshold)</p>
</blockquote>
</aside>
<pre data-code-wrap="python"><code class="lang-python">out = input.max(threshold)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

dlmacedo: <blockquote>
<pre><code> threshold = autograd.Variable(torch.rand(1), requires_grad=True)
 self.features = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=5, stride=1, padding=2),
    nn.Max(threshold),
    nn.MaxPool2d(kernel_size=3, stride=2),
    nn.Conv2d(64, 192, kernel_size=5, padding=2),
    nn.Max(threshold),
    nn.MaxPool2d(kernel_size=3, stride=2),
 )
</code></pre>
</blockquote>
<p>Do you think the code above should work?</p>

hughperkins: <p>As far as I can tell, it’s not possible. Also see <a href="https://discuss.pytorch.org/t/creating-a-custom-loss-function/4749/11?u=hughperkins">Creating a custom loss function</a></p>
<p>So, it looks like you could create a custom autograd module to handle this.  If it was me, I might consider logging it on pytorch issues page and/or submitting the custom autograd module for PR.</p>

albanD: <p>Hi,</p>
<p>The think is that the <code>threshold</code> operation is not differentiable wrt the threshold value.<br>
More specificatly, if the operation it is performing for each element is:</p>
<pre><code class="lang-auto">if inp[el] &lt;= threshold:
    out[el] = thresholded_value
else:
    out[el] = inp[el]
</code></pre>
<p>What is <code>d(out)/d(threshold)</code> here?</p>

hughperkins: <p>It works in tensorflow.  I reckon it’s not differentiable at the threshold itself, but it’s differnetaible almost everywhere?</p>

albanD: <p>Well the <code>threshold_value</code> will have a gradient that accumulate the <code>grad_out</code> for every element where it has been thresholded. So this one in theory you could learn, even though I am not sure what that means in practice.</p>
<p>The <code>threshold</code> is definitely not learnable with pure gradients, or maybe I am missing something? What would be the gradient “almost everywhere” ?</p>

hughperkins: <p>So, we have:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/e64b694a19911841a14ba0cd1c8b3cd2d249afff.png" data-download-href="https://discuss.pytorch.org/uploads/default/e64b694a19911841a14ba0cd1c8b3cd2d249afff" title="Screen Shot 2017-07-11 at 2.27.29 PM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/e64b694a19911841a14ba0cd1c8b3cd2d249afff_2_659x499.png" width="659" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/e64b694a19911841a14ba0cd1c8b3cd2d249afff_2_659x499.png, https://discuss.pytorch.org/uploads/default/optimized/1X/e64b694a19911841a14ba0cd1c8b3cd2d249afff_2_988x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/e64b694a19911841a14ba0cd1c8b3cd2d249afff.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-11 at 2.27.29 PM.png</span><span class="informations">1206×914 36.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

hughperkins: <p>(by the way, you can see that the theoretical result I’ve proposed matches the results I’m getting from tensorflow)</p>

tom: <p>Sum of the output_grad for things below the threshold, zero otherwise.<br>
You can see this by looking sternly at the max formulation or, if you prefer, rewrite as relu(x-t)+t.</p>
<p>Best regards</p>
<p>Thomas</p>

albanD: <p><a class="mention" href="/u/tom">@tom</a> good point when <code>threshold == threshold_value</code>.<br>
But can you get a similar expression for the general formula of threshold when they are not equal?</p>

tom: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>No, I do not know what to do then with respect to the input cut-off. And with my bias towards theory, a using discontinuous function seems unintuitive, too.<br>
In fact, I prefer to think about this as shrinkage (i.e. <code>relu(x-t)</code>, with its well-studied connections e.g. to a quadratic activation penalty or regression with noisy observation) plus a bias and don’t really like to think about thresholding. If you fed the output to (optionally) relu plus a layer that uses bias, I would think do not need the offset <code>+t</code> outside the relu at all.</p>
<p>But that’s me.</p>
<p>Best regards</p>
<p>Thomas</p>

hughperkins: <p>I’m like 80% sure it should be differentaible. What makes you feel it would not be?</p>
<p>By the way, one of the nice things about tf is, I’ve tried some really convoluted bizarre cost functions, and they’ve all been differentabiel. It’s quite nice…</p>

hughperkins: <p>Oh youre right. Fair enough <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/2258c5eef4745471e31d6f77a64d55f151a028a5.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/2258c5eef4745471e31d6f77a64d55f151a028a5" title="Screen Shot 2017-07-12 at 1.18.54 AM.jpg"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/2258c5eef4745471e31d6f77a64d55f151a028a5_2_540x500.jpg" width="540" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/2258c5eef4745471e31d6f77a64d55f151a028a5_2_540x500.jpg, https://discuss.pytorch.org/uploads/default/optimized/1X/2258c5eef4745471e31d6f77a64d55f151a028a5_2_810x750.jpg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/2258c5eef4745471e31d6f77a64d55f151a028a5_2_1080x1000.jpg 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-12 at 1.18.54 AM.jpg</span><span class="informations">1276×1180 57.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

dlmacedo: <p>What is wrong bellow?</p>
<blockquote>
<p>import torch<br>
from torch import autograd</p>
</blockquote>
<blockquote>
<p>import torch.nn as nn</p>
</blockquote>
<blockquote>
<p><strong>all</strong> = [<br>
‘CNN’, ‘cnn’,<br>
]</p>
</blockquote>
<blockquote>
<p>class CNN(nn.Module):<br>
def <strong>init</strong>(self, dataset):<br>
super(CNN, self).<strong>init</strong>()<br>
self.threshold = autograd.Variable(torch.rand(1), requires_grad=True)<br>
self.threshold.data.fill_(0.05)<br>
self.conv1 = nn.Conv2d(1, 10, kernel_size=5)<br>
self.conv2 = nn.Conv2d(10, 20, kernel_size=5)<br>
self.conv2_drop = nn.Dropout2d()<br>
self.relu = nn.ReLU(inplace=True)<br>
self.maxpool = nn.MaxPool2d(kernel_size=2, stride=None, padding=0)<br>
self.fc1 = nn.Linear(320, 50)<br>
self.fc2 = nn.Linear(50, 10)</p>
</blockquote>
<blockquote>
<pre><code>def forward(self, x):
    x = self.conv1(x)
    x = self.maxpool(x)
    #x = x + self.threshold.expand_as(x)
    x = x + self.threshold
    x = self.relu(x)
    x = self.conv2(x)
    x = self.conv2_drop(x)
    x = self.maxpool(x)
    x = self.relu(x)
    x = x.view(-1, 320)
    x = self.fc1(x)
    x = self.relu(x)
    x = self.fc2(x)
    x = self.relu(x)
    return x
</code></pre>
</blockquote>
<blockquote>
<p>def cnn(dataset):<br>
model = CNN(dataset)<br>
return model</p>
</blockquote>
<p>Error:</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 390, in <br>
main()<br>
File “train.py”, line 165, in main<br>
cnn(args.epochs, train_loader, val_loader, model, criterion, optimizer, experiment)<br>
File “train.py”, line 200, in cnn<br>
training_time += train(train_loader, model, criterion, optimizer, epoch)<br>
File “train.py”, line 260, in train<br>
output = model(input_var)<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 206, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 59, in forward<br>
return self.module(*inputs[0], **kwargs[0])<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 206, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/dlm/code/deeplearninglab/sem/models/cnn.py”, line 27, in forward<br>
x = x + self.threshold<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py”, line 745, in <strong>add</strong><br>
return self.add(other)<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py”, line 283, in add<br>
return self._add(other, False)<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py”, line 277, in _add<br>
return Add(inplace)(self, other)<br>
File “/home/dlm/anaconda3/lib/python3.6/site-packages/torch/autograd/_functions/basic_ops.py”, line 20, in forward<br>
return a.add(b)<br>
TypeError: add received an invalid combination of arguments - got (torch.FloatTensor), but expected one of:</p>
<ul>
<li>(float value)<br>
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)</li>
<li>(torch.cuda.FloatTensor other)<br>
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)</li>
<li>(torch.cuda.sparse.FloatTensor other)<br>
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)</li>
<li>(float value, torch.cuda.FloatTensor other)</li>
<li>(float value, torch.cuda.sparse.FloatTensor other)</li>
</ul>

tom: <p>I’d use <code>nn.Parameter</code> instead of <code>Variable</code> for the Parameter. You did call <code>model.cuda</code>, probably.</p>
<p>Best regards</p>
<p>Thomas</p>

albanD: <p>You should do <code>self.threshold = nn.Parameter(torch.rand(1))</code>.<br>
All parameters of a <code>nn.Module</code> must be <code>nn.Parameter</code>s otherwise they won’t appear when you call <code>.parameters()</code> and won’t move when you call <code>.cuda()</code> (which is your problem here).</p>

dlmacedo: <pre><code>class DDReLU(nn.Module):
    def __init__(self):
        super(DDReLU, self).__init__()
        self.threshold = nn.Parameter(torch.rand(1), requires_grad=True)
        self.register_backward_hook(lambda module, grad_i, grad_o: (grad_i[0], grad_i[1]*0.01))
        #self.threshold.data.fill_(0.1)
        self.ReLU = nn.ReLU(True)

    def forward(self, x):
        print(self.threshold.data[0])
        return self.ReLU(x + self.threshold) - self.threshold
        #return self.ReLU(x) + self.threshold
</code></pre>
<p>Is the code above fine to change the relative learning rate of the new parameter?</p>
<p>By relative learning rate, I mean: The parameter created has a learning rate that is 0.01 times the one used to the other model’s parameters.</p>

sparseinference: <p>I’ve been experimenting with learning the threshold parameters for expressions like  <strong>.clamp(min=lower)</strong> where ‘<strong>lower</strong>’ is a Module Parameter.</p>
<p>Here is a function for accomplishing it for clamping to zero or negative values:</p>
<pre><code>def Clamp(x, minval):
    """
    Clamps Variable x to minval.
    minval &lt;= 0.0
    """
    return x.clamp(max=0.0).sub(minval).clamp(min=0.0).add(minval) + x.clamp(min=0.0)
</code></pre>
<p>With some extra work the same could be done for  <strong>.clamp(max=upper)</strong> .</p>