Differentiable torch.histc?

AShibaInu: <p>Hi everyone, I found out that the derivative for ‘histc’ is not implemented. Is there a way to easily implement a histogram function with derivates?</p>
<p>Thank you for your time</p>

Rishi_Rawat: <p>Heres’s something that might fill the need:</p>
<pre><code class="lang-auto">class HistoBin(nn.Module):
    def __init__(self, locations = np.arange(0,1,.1), radius=.2, norm = True):
        super(HistoBin, self).__init__()
        
        self.locs = locations
        self.r = radius
        self.norm = norm
    
    def forward(self, x):
        
        counts = []
        
        for loc in self.locs:
            dist = torch.abs(x - loc)
            #print dist
            ct = torch.relu(self.r - dist).sum(1) 
            counts.append(ct)
        
        out = torch.stack(counts, 1)
        
        if self.norm:
            summ = out.sum(1) + .000001
            return (out.transpose(1,0) / summ).transpose(1,0)
        return out
</code></pre>

Tony-Y: <p><a href="https://www.nature.com/articles/ncomms13890" rel="nofollow noopener">Gaussian expansion</a> could be used as an approximate histogram:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

data = 50 + 25 * torch.randn(1000)

hist = torch.histc(data, bins=10, min=0, max=100)

print(hist)

class GaussianHistogram(nn.Module):
    def __init__(self, bins, min, max, sigma):
        super(GaussianHistogram, self).__init__()
        self.bins = bins
        self.min = min
        self.max = max
        self.sigma = sigma
        self.delta = float(max - min) / float(bins)
        self.centers = float(min) + self.delta * (torch.arange(bins).float() + 0.5)

    def forward(self, x):
        x = torch.unsqueeze(x, 0) - torch.unsqueeze(self.centers, 1)
        x = torch.exp(-0.5*(x/self.sigma)**2) / (self.sigma * np.sqrt(np.pi*2)) * self.delta
        x = x.sum(dim=1)
        return x

gausshist = GaussianHistogram(bins=10, min=0, max=100, sigma=6)

data.requires_grad = True
hist = gausshist(data)
print(hist)

hist.sum().backward()
print(data.grad)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([ 41.,  63.,  98., 123., 145., 158., 128., 106.,  67.,  29.])
tensor([ 38.4469,  65.1605,  92.4981, 122.9916, 145.3853, 150.2217, 130.5083,
        102.5501,  69.8868,  31.8439], grad_fn=&lt;SumBackward2&gt;)
tensor([ 6.4244e-02, -7.6090e-02,  5.6231e-05,  9.3874e-04, -7.5421e-04,
        -2.4916e-04, -1.0152e-03,  6.8575e-04,  1.8976e-04,  6.6737e-04,
        -7.3137e-05,  4.9042e-02,  1.8404e-04, -2.6847e-04, -5.8353e-04,
</code></pre>

Tony-Y: <p>The soft histogram might be usable instead of <code>torch.histc</code>.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

data = 50 + 25 * torch.randn(1000)

hist = torch.histc(data, bins=10, min=0, max=100)

print(hist)

class SoftHistogram(nn.Module):
    def __init__(self, bins, min, max, sigma):
        super(SoftHistogram, self).__init__()
        self.bins = bins
        self.min = min
        self.max = max
        self.sigma = sigma
        self.delta = float(max - min) / float(bins)
        self.centers = float(min) + self.delta * (torch.arange(bins).float() + 0.5)

    def forward(self, x):
        x = torch.unsqueeze(x, 0) - torch.unsqueeze(self.centers, 1)
        x = torch.sigmoid(self.sigma * (x + self.delta/2)) - torch.sigmoid(self.sigma * (x - self.delta/2))
        x = x.sum(dim=1)
        return x

softhist = SoftHistogram(bins=10, min=0, max=100, sigma=3)

data.requires_grad = True
hist = softhist(data)
print(hist)

hist.sum().backward()
print(data.grad.max())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([ 27.,  59.,  89., 134., 153., 183., 142.,  97.,  52.,  32.])
tensor([ 26.2054,  60.0647,  87.4429, 135.9337, 150.7281, 184.4028, 139.8741,
         97.9176,  52.9988,  30.5099], grad_fn=&lt;SumBackward2&gt;)
tensor(0.7499)
</code></pre>

Peter_Featherstone: <p>This function only seems to work for your input. For example, input values between -1 and 1 don’t work</p>

Peter_Featherstone: <p>In fact, it only works for the specific data and parameters provided</p>

Tony-Y: <p>Please adjust parameters to your data.</p>

Peter_Featherstone: <p>Well there is only one parameter, sigma. I’ve tried numerous. None work. I welcome you to try. Simply replace “data = 50 + 25 * torch.randn(1000)” with “data = torch.randn(1000)”. I’m guessing neither gaussianhist or softhist generalise particularly well and require input data to have a distribution resembling a gaussian one.</p>

Tony-Y: <p>Please try min=-2, max=2, sigma=3/25. I did shift and scale the parameters to adjust them to the normal data.</p>

Peter_Featherstone: <aside class="quote no-group" data-username="Tony-Y" data-post="4" data-topic="25865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tony-y/48/8373_2.png" class="avatar"> Tony-Y:</div>
<blockquote>
<p>softhist = SoftHistogram(bins=10, min=0, max=100, sigma=3)</p>
</blockquote>
</aside>
<p>Thanks for the reply. Unfortunately, that doesn’t work. That was the first thing I tried. It works even less so when you increase bins.</p>

Tony-Y: <p>I am sorry. The sigma should be 3*25.</p>

Tony-Y: <p>When min=-2, max=2, sigma=3*25, I got the following:</p>
<pre><code class="lang-auto">tensor([ 36.,  72., 105., 136., 134., 148., 132.,  92.,  55.,  37.])
tensor([ 36.3019,  71.9146, 104.1209, 138.0593, 133.3760, 144.7053, 134.9035,
         92.9224,  52.8923,  36.8525], grad_fn=&lt;SumBackward2&gt;)
tensor(18.4180)
</code></pre>
<p>The parameter name “sigma” is a bit confusing. I think “kappa” or “k” is better.</p>

Peter_Featherstone: <p>Yep it’s getting there. Now set bins=255.</p>

Peter_Featherstone: <p>I get tensor([ 0.,  0.,  0.,  2.,  0.,  1.,  1.,  1.,  2.,  0.,  1.,  0.,  0.,  0.,<br>
1.,  1.,  1.,  1.,  1.,  2.,  3.,  0.,  2.,  1.,  1.,  2.,  0.,  2.,<br>
1.,  2.,  3.,  0.,  0.,  2.,  1.,  0.,  3.,  4.,  2.,  3.,  1.,  2.,<br>
4.,  2.,  1.,  1.,  4.,  2.,  5.,  1.,  4.,  3.,  3.,  4.,  0.,  2.,<br>
5.,  6.,  4.,  3.,  8.,  1.,  4.,  5.,  3.,  0.,  3.,  5.,  4.,  4.,<br>
2.,  6.,  3.,  7.,  2., 10.,  2.,  5., 10.,  3.,  7.,  6.,  6.,  7.,<br>
4.,  3.,  2.,  7.,  4.,  7.,  3.,  4.,  5.,  5.,  5.,  2.,  3.,  6.,<br>
3.,  9.,  6.,  7.,  7.,  6.,  5.,  7.,  5.,  6.,  6.,  5.,  7.,  3.,<br>
5.,  4.,  2.,  7.,  2., 10.,  4.,  5., 11., 11.,  5.,  9.,  6.,  7.,<br>
4.,  7.,  6.,  9.,  7., 11.,  6.,  7.,  7.,  5.,  6.,  5.,  6.,  4.,<br>
9.,  8.,  7., 12.,  9.,  8.,  1.,  6.,  4.,  5.,  5.,  6.,  5.,  7.,<br>
11., 11.,  4.,  7.,  9.,  3.,  4.,  6.,  4.,  3.,  4., 10.,  3.,  2.,<br>
6., 11.,  4.,  4.,  6.,  7.,  4.,  5.,  5.,  3.,  3.,  3.,  5.,  4.,<br>
8.,  2.,  4.,  4.,  2.,  9.,  6.,  4.,  4.,  2.,  3.,  6.,  3.,  2.,<br>
2.,  5.,  3.,  0.,  2.,  1.,  6.,  2.,  3.,  0.,  6.,  4.,  0.,  4.,<br>
1.,  0.,  2.,  3.,  2.,  5.,  1.,  2.,  5.,  2.,  2.,  1.,  3.,  4.,<br>
1.,  5.,  2.,  2.,  1.,  4.,  3.,  2.,  3.,  3.,  1.,  0.,  4.,  0.,<br>
1.,  0.,  0.,  2.,  2.,  0.,  1.,  0.,  0.,  2.,  1.,  1.,  0.,  2.,<br>
0.,  2.,  0.])<br>
tensor([0.2294, 0.2264, 0.4174, 0.6920, 0.8129, 0.8522, 0.9543, 1.0392, 1.0053,<br>
0.8288, 0.6072, 0.4167, 0.3240, 0.4156, 0.6070, 0.8465, 1.0836, 1.1995,<br>
1.3867, 1.6308, 1.6203, 1.4406, 1.3543, 1.2456, 1.1760, 1.1920, 1.1754,<br>
1.2341, 1.4459, 1.7166, 1.6359, 1.1794, 0.9422, 1.0185, 1.1744, 1.5338,<br>
2.1595, 2.6610, 2.7007, 2.4503, 2.2206, 2.2761, 2.3876, 2.1789, 1.9693,<br>
2.1244, 2.4716, 2.7490, 2.9155, 2.9682, 3.0545, 3.1094, 3.0652, 2.7475,<br>
2.4773, 2.9287, 3.8287, 4.4061, 4.3883, 4.2728, 4.3133, 4.1353, 3.9032,<br>
3.5445, 3.0214, 2.7678, 3.0349, 3.4247, 3.5377, 3.5801, 3.9689, 4.5266,<br>
4.7314, 4.8329, 5.1060, 5.3357, 5.2748, 5.6173, 6.1142, 6.1377, 6.0390,<br>
6.0027, 5.9284, 5.4285, 4.6102, 4.0881, 4.1167, 4.4975, 4.7918, 4.8103,<br>
4.5924, 4.5281, 4.7080, 4.6561, 4.1667, 3.7260, 3.9051, 4.5131, 5.3062,<br>
6.2507, 6.7920, 6.8175, 6.6299, 6.2593, 5.9633, 5.8159, 5.6079, 5.4935,<br>
5.5841, 5.6613, 5.3981, 4.9287, 4.5209, 4.0846, 3.9914, 4.4455, 5.1549,<br>
5.9601, 6.4060, 7.0180, 8.1164, 8.5827, 8.1205, 7.4650, 6.5912, 5.9549,<br>
5.8980, 6.1648, 6.8313, 7.6569, 8.1927, 8.1415, 7.4805, 6.9542, 6.6359,<br>
6.1840, 5.7488, 5.6370, 5.8313, 6.2864, 7.1439, 7.9608, 8.6149, 8.8756,<br>
8.0024, 6.5094, 5.2868, 4.8778, 4.9382, 5.0370, 5.1563, 5.5232, 6.3218,<br>
7.4457, 8.2003, 8.1334, 7.5174, 7.0089, 6.4302, 5.5020, 4.8912, 4.7707,<br>
4.6245, 4.5781, 4.8791, 5.0248, 4.6596, 4.7419, 5.6951, 6.3835, 6.0732,<br>
5.7131, 5.5538, 5.1487, 4.7616, 4.6408, 4.3855, 3.9211, 3.7476, 3.9929,<br>
4.3637, 4.6391, 4.6718, 4.3453, 4.0791, 4.0660, 4.5383, 5.3572, 5.4346,<br>
4.6449, 3.8484, 3.4765, 3.6354, 3.7055, 3.2702, 2.8501, 2.7852, 2.8737,<br>
2.6213, 2.1645, 2.0984, 2.5559, 2.9686, 2.8456, 2.6406, 2.8016, 3.1273,<br>
2.9296, 2.4468, 2.0567, 1.5683, 1.3667, 1.7574, 2.3771, 2.7778, 2.8132,<br>
2.7282, 2.8660, 2.8734, 2.4603, 2.0452, 2.0514, 2.4907, 2.9116, 3.0435,<br>
2.9171, 2.5364, 2.2664, 2.3686, 2.6276, 2.6501, 2.5337, 2.4915, 2.2262,<br>
1.7326, 1.5266, 1.5451, 1.2724, 0.8348, 0.6324, 0.7961, 1.0441, 1.0182,<br>
0.7880, 0.6156, 0.5596, 0.7096, 0.9891, 1.0293, 0.8919, 0.8464, 0.9157,<br>
0.9453, 0.8344, 0.6318])</p>

Tony-Y: <p>There may be some errors of about 1 because of the softening. Keep in mind that Heaviside step functions in histogram are replaced with sigmoid functions.</p>

Peter_Featherstone: <p>Yeah. i don’t think this is a solvable problem. You can sharpen the sigmoid function, but in doing so you’re going to increase your gradients. As your “sharpened” sigmoid approaches the heavy-side function, your gradients are going to diverge. I think histograms are a lost cause unless you use them as the very first input of your model, where tensors don’t require gradients.</p>

Tony-Y: <p>You are right. Actually, I didn’t use histograms in any neural network, and don’t want to use them. I just created the soft histogram in analogy with the soft max, focusing on the softening.</p>

Peter_Featherstone: <p>It’s a shame though. Histograms are such useful features. Ideal to plug into your final dense layer. Oh well</p>

Tony-Y: <p>I hope your attempt succeeds.<br>
Good luck!</p>

Yihui_Ren: <p>Hi <a class="mention" href="/u/tony-y">@Tony-Y</a></p>
<p>Thank you for your code snippet.  It is very helpful.<br>
I have one suggestion:<br>
In order to make <code>model.cuda()</code> work, we need to wrap the <code>self.centers</code> with <code>nn.Parameter()</code> like:</p>
<pre><code class="lang-auto">self.centers = nn.Parameter(self.centers, requires_grad=False)
</code></pre>
<p>Also, if we are expecting the first dimension is the batch size, I update the<br>
<code>forward()</code> function as follows:</p>
<pre><code class="lang-auto">
    def forward(self, x):
        x = torch.unsqueeze(x, 1) - torch.unsqueeze(self.centers, 1)
        x = torch.exp(-0.5*(x/self.sigma)**2) / (self.sigma * np.sqrt(np.pi*2)) * self.delta
        x = x.sum(dim=-1)
        x = x/x.sum(dim=-1).unsqueeze(1) # normalization
        return x
</code></pre>