Add "sincos" layers in the nn

username: DanielC
cooked: <p>I am wondering if there is existing function allow us to add two nodes sin and cos in a net like an activation function?</p>

username: ptrblck
cooked: <p>I’m not sure I understand the use case correctly, but in case you want to apply <code>torch.cos</code> and <code>torch.sin</code> on an activation you could directly do it via:</p>
<pre><code class="lang-python">def forward(self, x): # forward method of your model
    ...
    x = self.layer(x)
    ...
    x = torch.sin(x)
    x = torch.cos(x)
    ...
    return x
</code></pre>
<p>or alternatively you could also create a custom <code>nn.Module</code> and use these operations as an “activation” function.</p>

username: DanielC
cooked: <p>Thank you ptrblck, let me explain my question more clarify. I am implementing a GAN, in order to endure the generator output to be periodic. I want to add a “layer” as the last layer of generator to change x into (sin(x), cos(x)). I want this “layer” is part of of backprop as well. since the dimension is changed, so I use <code>torch.cat</code> to solve the problem:</p>
<pre><code class="lang-python">    def forward(self, noise):
        angle = self.gen(noise)
        output = torch.cat((torch.sin(angle), torch.cos(angle)))
        return output
</code></pre>
<p>However there are two question:</p>
<ol>
<li>if the <code>torch.cat()</code> function is allowed to do backprop? I find the value of <code>grad_fn</code> become to <code>&lt;CatBackward&gt;</code>
</li>
<li>I checked the output of generator after adding the <code>output</code> line in forward function, but didn’t see any change. I must miss something about the <code>forward</code> function. could you please give me some guide (or a link) so I can figure it out.</li>
</ol>
<p>Thank you very much !!</p>

username: ptrblck
cooked: <ol>
<li>Yes, <code>torch.cat</code> has a valid <code>grad_fn</code> and will not break the backpropagation.</li>
<li>What did you compare the outputs against? Your <code>forward</code> looks good and the added operations should be used. You could add additional <code>print</code> statements to the <code>forward</code> to make sure it’s really called and to check intermediate outputs.</li>
</ol>

username: hiba_younis
cooked: <p>Hello<br>
How I can add a sinc convolution 1d layer before a pretrained wav2vec2xlsr model for ASR  to enhance the model.<br>
thanks in advance</p>

username: ptrblck
cooked: <p>You might be able to call into your custom layer before passing its output to the pretrained model. Alternatively, you could also try to add the custom layer directly into your model e.g. by replacing the first layer with an <code>nn.Sequential</code> block containing the new layer as well as the original one.</p>

username: hiba_younis
cooked: <p>Could you explain it more with example about add sinc conv1d layer to wav2vec2xlsr model, I am new in this field.<br>
I appreciate any help.</p>
<p>في السبت، ١١ نوفمبر ٢٠٢٣ ٧:١٣ م ptrblck via PyTorch Forums &lt;<a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a>&gt; كتب:</p>

username: vadimkantorov
cooked: <p>Related discussion: <a href="https://github.com/pytorch/pytorch/issues/90559" class="inline-onebox" rel="noopener nofollow ugc">[feature request, idea] Fused torch.sincos(x) or cossin(x) (maybe alias to complex exponential) · Issue #90559 · pytorch/pytorch · GitHub</a></p>