akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there any pretrained model for nn.Transformer?

Magolor: <p>Papers have shown that NLP models pretrained on any dataset (even not language, e.g.: music, codes, brackets, etc) can improve the performance on other datasets.<br>
So is there any pretrained nn.Transformer parameters available? I would really appreciate it if anyone could provide one. Specifically, I’m doing a music-related task, so it would be super amazing to find a nn.Transformer model pretrained on music data.</p>

Abhilash_Srivastava: <p>I usually refer to the HuggingFace pre-trained models. You can refer <a href="https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling" rel="noopener nofollow ugc">here</a>.</p>
<p>I had worked on a music generation project (not directly relying on pre-trained models though).<br>
My guess is that pre-trained models (like BERT, Roberta) may not perform well on music-related tasks because they weren’t trained on music data (rather on more general datasets like Wiki, Book corpus). It is still worth a try though.</p>

Magolor: <p>Thanks, but it seems that the Hugging Face provides highly integrated models for language-related tasks only. It involves predefined and pretrained tokenizers and other sort of things. Could I possibly extract the parameters of the models?</p>

Abhilash_Srivastava: <p>Yes, these pre-trained models are trained for language tasks. So, if you have to use them, you’ll need to create your own vocabulary and tokenizer. A better idea may be to just train the transformer model yourself. Even if you go extract the pre-trained weights and load it into your model, I don’t think it will help a lot.</p>
<p>I remember I found some useful resources on the Google Magenta <a href="https://magenta.tensorflow.org/" rel="noopener nofollow ugc">website</a>.</p>
Weighted Batch Matrix Multiplication

eugeny: <p>Hello,</p>
<p>v1 : BatchSize x MaxSentenceLength x EmbSize<br>
v2 : BatchSize x EmbSize x MaxSentenceLength</p>
<p>and</p>
<p>v3 = torch.bmm(v1, v2) : BatchSize x MaxSentenceLength x MaxSentenceLength</p>
<p>As I understand - torch.bmm based on simple dot product of 2 two embeddings:</p>
<pre><code>                               sum(emb1[i] * emb2[i],  i = [0..EmbSize])
</code></pre>
<p>But for me interesting version of torch.bmm based on weighted dot product of 2 embeddings:</p>
<pre><code>                               sum(w[i] * emb1[i] * emb2[i],  i = [0..EmbSize])
</code></pre>
<p>where  w[i] are learned weights.</p>
<p>Is there a way to do this ?</p>
<p>Thanks in advance!</p>

InnovArul: <p>If I understand correctly, <code>w</code> is learnable parameter with size <code>EmbSize</code>.<br>
What if you can do the following?</p>
<pre><code class="lang-auto">BatchSize = 10
MaxSentenceLength = 20
EmbSize = 30

v1 = torch.randn(BatchSize, MaxSentenceLength, EmbSize)
v2 = torch.randn(BatchSize, EmbSize, MaxSentenceLength)
w = nn.Parameter(torch.randn(1,1,EmbSize))

out = torch.bmm(w * v1, v2)

</code></pre>