My transform model

username: slavavs
cooked: <p>I am trying to do NN based on this article <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="noopener nofollow ugc">http://nlp.seas.harvard.edu/2018/04/03/attention.html</a><br>
Or based on the standard nn.transformer module<br>
I have numbers instead of words. The size of one sequence is 5. At the output(I have three neurons), I need to predict three values ​​(0, 1, 2). That is, at each step I take a window of size 5 and predict 0 or 1 or 2. In fact, this is a classification into 3 classes. Can I use the standard nn.transformer module and use my data instead of words? Or what do you advise me to do?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/82f0327e32fad85f16b9f20284b16a140aedf859.png" data-download-href="https://discuss.pytorch.org/uploads/default/82f0327e32fad85f16b9f20284b16a140aedf859" title="transformer"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/82f0327e32fad85f16b9f20284b16a140aedf859.png" alt="transformer" data-base62-sha1="iGkGRChsFfKyjnF9e4YTQO0bCo9" width="690" height="437" data-dominant-color="EBEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">transformer</span><span class="informations">972×616 6.53 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: slavavs
cooked: <p>If you consider my example based on the nn.transformer module<br>
Instead of words, I use a vector of numbers of size 7. I want to send this vector instead of the word directly to the nn.transformer module<br>
The size of my sequence is always 5.<br>
How can I make a simple example to check the functionality of the code? I do not understand what E. is</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Trans(nn.Module):
    def __init__(self, src_vocab, tgt_vocab):
        super(Trans, self).__init__()
        self.tr = nn.Transformer(src_vocab, tgt_vocab)
        
    def forward(self, src, tgt):
        out = self.tr(src, tgt)
        
        return out
    
net = Trans(5,1) #src=5 (window), tgt=1 ('0' or '1' or '2')

sr = torch.randn(5, 1, 7) #src: (S, N, E) E-??
tg = torch.tensor([0], dtype=torch.long) #tgt: (T, N, E)
outputs = net(sr, tg) #output: (T, N, E)
</code></pre>

username: slavavs
cooked: <p>Another addition to my question.<br>
If my question is not clear, you tell me. I would like to get a very simple example of how to solve this so that I can apply this with my real example.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/977f451b96ac592ea906f4e652a3276a8923eba8.png" data-download-href="https://discuss.pytorch.org/uploads/default/977f451b96ac592ea906f4e652a3276a8923eba8" title="transformer2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/977f451b96ac592ea906f4e652a3276a8923eba8_2_690x205.png" alt="transformer2" data-base62-sha1="lCcLU4OsenTMbvT6Jmtt9tMB6vm" width="690" height="205" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/977f451b96ac592ea906f4e652a3276a8923eba8_2_690x205.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/977f451b96ac592ea906f4e652a3276a8923eba8_2_1035x307.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/977f451b96ac592ea906f4e652a3276a8923eba8_2_1380x410.png 2x" data-dominant-color="F4F4F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">transformer2</span><span class="informations">1967×587 10.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>