Multi-wavelet activation function

William_yang: <p>Hi, I am a starter of Pytorch. I want to implement a multi-wavelet function as an activation function. However, I don’t know how to set a function in this module.<br>
`class MultiWavelet(nn.Module):</p>
<pre><code>def __init__(self,out_features):
    super(MultiWavelet, self).__init__()
    self.out_features = out_features
    self.scale = nn.Parameter(torch.Tensor(1,self.out_features),requires_grad = True)
    self.shift = nn.Parameter(torch.Tensor(1,self.out_features),requires_grad = True)
    self.a = nn.Parameter(torch.Tensor(1,self.out_features),requires_grad = True)
    self.b = nn.Parameter(torch.Tensor(1,self.out_features),requires_grad = True)
    
def func1(self,c):
    self.c = c
    out = self.c.detach().numpy().copy()
    out[out&gt;=0 &amp; out &lt;1] = -2 * torch.pow(out,3) + 3 * torch.pow(out,2)
    out[out&gt;=1 &amp; out &lt;=2] = torch.pow((2-out),2) * (2*out - 1)
    out[out &lt;0 &amp; out&gt;2] = 0
    return out
    

def Wavelet2(self,c):
    self.c = c
    out = self.c.detach().numpy().copy()
    out[out&gt;=0 &amp; out &lt;1] = -torch.pow(out,2) * (3 * out - 3)
    out[out&gt;=1 &amp; out &lt;=2] = -torch.pow((2-out),2) * (3* out - 3)
    out[out &lt;0 &amp; out&gt;2] = 0
    return out

def forward(self, x):
    x_out = (x-self.shift)/self.scale
    h1 = self.Wavelet1(x_out)
    h2 = self.Wavelet(x_out)
    h = a. * h1 + b. * h2
    
    return h`
</code></pre>
<p>In this case, I get a runtime error.That is TypeError: pow() received an invalid combination of arguments - got (numpy.ndarray, int), but expected one of:</p>
<ul>
<li>(Tensor input, Tensor exponent, Tensor out)</li>
<li>(Number self, Tensor exponent, Tensor out)</li>
<li>(Tensor input, Number exponent, Tensor out)</li>
</ul>
<p>Thank you all in advance.</p>

ptrblck: <p>Could you remove the <code>numpy()</code> call from <code>self.c</code> and rerun the code?</p>
<p>Generally, if you don’t need to use another lib, I would recommend to use PyTorch methods only.<br>
This will make sure that Autograd is able to keep track of all operations and create the backward pass for you automatically.<br>
If you need to use another lib such as numpy, you would need to implement the <code>backward</code> function manually.</p>
<p>That being said, the current error is thrown, since <code>out</code> is a numpy array and you are trying to apply <code>torch.pow</code> on it.</p>

William_yang: <p>Thank you for your reply.<br>
I tried to remove the <code>numpy()</code> call from <code>self.c</code>, and it worked in the function. But there are some other problems such as Tensor could not use the <code>copy()</code>. So I remove the <code>copy()</code>, too. But for the matrix data processing, just like</p>
<pre><code class="lang-auto">b = torch.randn(2,2)*1.5
a = b.detach()
a[(a&gt;=0) &amp; (a &lt;1)] = -2 * torch.pow(a,3) + 3 * torch.pow(a,2)
</code></pre>
<p>It is a RuntimeError.</p>
<pre><code class="lang-auto">RuntimeError: shape mismatch: value tensor of shape [2, 2] cannot be broadcast to indexing result of shape [1]
</code></pre>
<p>Is there a better grammar in torch could solve this problem.</p>
<p>Thank you for your time.</p>

ptrblck: <p>The left-hand side will return a variable number of elements (it could be empty, contain some elements, or all), while the right-hand side is a tensor in the shape <code>[2, 2]</code>, which won’t work.<br>
Could you explain, what you are trying to achieve?</p>

William_yang: <p>Dear Ptrblck,</p>
<p>I am so sorry for I did not explain well.<br>
I want to try this function through Pytorch.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/6117922a7209fee9cbe3ec65965b72dcfd5a86c5.png" alt="image" data-base62-sha1="dQUOxyT52wvKvo4nzrXiTGfejqJ" width="296" height="139"><br>
So in a matrix, I want to classify numbers of different size ranges.<br>
In this case, if I still don’t explain well, could you let me know? I will try again.</p>
<p>Thank you for your time.</p>

William_yang: <p>For example, in this a matrix, <code>a[(a&gt;=0)&amp; (a&lt;1)]</code> , I could get a tensor <code>tensor([0.5340])</code>. So I want this number could transform into <code>-2 * torch.pow(tensor([0.5340]),3) + 3 * torch.pow(tensor([0.5340]),2)</code> at the same place of this matrix.</p>

ptrblck: <p>Thanks for the clarification.<br>
In that case, this might work:</p>
<pre><code class="lang-python">b = torch.randn(2,2)*1.5
a = b.detach()
idx = (a&gt;=0) &amp; (a &lt;1)
if idx.any():
    a[idx] = -2 * torch.pow(a[idx],3) + 3 * torch.pow(a[idx],2)
</code></pre>

William_yang: <p>Dear Ptrblck,</p>
<p>Thanks for your reply.<br>
I try to use this code and get the right result. Is it the right grammar in Pytroch?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/424eb3a5ae94ccb729255ff2d90ed31b5d6e2df8.png" alt="image" data-base62-sha1="9sA86ERXF1owLIqS3TnlTUK6Ny8" width="618" height="281"></p>
<p>Thanks for your time.</p>

William_yang: <p>Dear ptrblck,</p>
<p>I have another problem with this function. Do you think the activation function I built could be auto-grad?</p>
<pre><code class="lang-auto">class MultiWavelet(nn.Module):

    def __init__(self,out_features):
        super(MultiWavelet, self).__init__()
        self.out_features = out_features
        self.scale = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
        self.shift = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
        self.a = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
        self.b = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
        
    def Wavelet1(self,c):
        self.c = c
        out = self.c.detach()
        out[(out&gt;=0) &amp; (out &lt;1)] = -2 * torch.pow(out[(out&gt;=0) &amp; (out &lt;1)],3) + 3 * torch.pow(out[(out&gt;=0) &amp; (out &lt;1)],2)
        out[(out&gt;=1) &amp; (out &lt;=2)] = torch.pow((2-out[(out&gt;=1) &amp; (out &lt;=2)]),2) * (2*out[(out&gt;=1) &amp; (out &lt;=2)] - 1)
        out[(out &lt;0) | (out&gt;2)] = 0
        return out
        
    
    def Wavelet2(self,c):
        self.c = c
        out = self.c.detach()
        out[(out&gt;=0) &amp; (out &lt;1)] = -torch.pow(out[(out&gt;=0) &amp; (out &lt;1)],2) * (3 * out[(out&gt;=0) &amp; (out &lt;1)] - 3)
        out[(out&gt;=1) &amp; (out &lt;=2)] = -torch.pow((2 - out[(out&gt;=1) &amp; (out &lt;=2)]),2) * (3* out[(out&gt;=1) &amp; (out &lt;=2)] - 3)
        out[(out&lt;0) | (out&gt;2)] = 0
        return out
    
    def forward(self, x):
        x_out = (x-self.shift)/self.scale
        h1 = self.Wavelet1(x_out)
        h2 = self.Wavelet2(x_out)
        h = self.a * h1 + self.b * h2
        
        return h
</code></pre>
<p>I need to compute these parameters, such as scale, shift, a, b, etc.</p>
<p>Look forward to hearing from you.<br>
Best regards.</p>

ptrblck: <p>Since you are detaching <code>c</code>, the computation graph will be broken at this point, which means that <code>self.shift</code> and <code>self.scale</code> shouldn’t get any gradients.<br>
Could you try to use <code>.clone()</code> instead or why are you detaching <code>c</code> at all? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

William_yang: <p>Thanks for your patience. It is my careless that I don’t understand <code>.detach()</code> well.</p>
<p>And if I set the parameters like this,</p>
<pre><code class="lang-auto">self.scale = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
self.shift = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
self.a = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
self.b = nn.Parameter(torch.randn(1,self.out_features),requires_grad = True)
</code></pre>
<p>I get the result like this.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea2cf87c5be73f05b3e79b34baa9e2dd83cb11bf.png" alt="image" data-base62-sha1="xpC4MXAgYeeFKkd7q9ZtKttsWmP" width="594" height="254"><br>
My loss functions set like this.</p>
<pre><code class="lang-auto">loss_func1 = nn.MSELoss()
loss_func2 = nn.CrossEntropyLoss()
loss1 = loss_func1(decoded, b_y.float())      # mean square error
loss2 = loss_func2(out, labels)
loss = loss1 + loss2
</code></pre>
<p>What are the possible reasons for this phenomenon? Gradient vanishing?</p>
<p>Thank you for your time.</p>

ptrblck: <p>Could you run your code with <code>torch.autograd.set_detect_anomaly(True)</code> and narrow down the operation, which creates the NaN values?<br>
Also, could you check the input data for invalid values?</p>