Input to transformer as linear or matrix

Flock1: <p>Hi,</p>
<p>I am very new to transformers and I am working with FNet by <a href="https://github.com/rishikksh20/FNet-pytorch" rel="noopener nofollow ugc">Rishikksh20</a>.</p>
<p>I was looking at some videos about transformers and I see that the input after embeddings is usually a matrix. However, in this implementation, there’s a <code>Linear</code> layer instead of 2d Convolution. So is the input usually a vector or is there a way I can input a matrix? I am doing this:</p>
<pre><code class="lang-auto">data_model = PositionalEncoding(data_model)
data_model = data_model.flatten()
</code></pre>
<p>Let me know if I’m doing it correctly.</p>

AlphaBetaGamma96: <p>The <code>nn.Linear</code> layer can take any input sized Tensor as can be seen in the docs (<a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear" class="inline-onebox" rel="noopener nofollow ugc">Linear — PyTorch 1.9.0 documentation</a>), the input has to be of shape <code>(N,∗,H)</code> where <code>H</code> is the number of input nodes in the layer, and <code>*</code> represent other dims. Find the shape of data_model (via <code>data_model.shape</code>) and the final dim is the number of nodes you need. (found via <code>data_model.shape[-1]</code>)</p>

Flock1: <p>So I am little confused with the implementation of positional embedding by PyTorch <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">here</a></p>
<p>Let’s say I have an embedding of dimension 10 and I have 14 such embeddings. So my matrix will be something like <code>(14X10)</code>. Now, if I want to get positional embedding of this matrix, I create an object and find the position embedding like this:</p>
<pre><code class="lang-auto">embed = PositionalEncoding(d_model=1)
res = embed(data)
</code></pre>
<p>but when I look at the shape, it’s this</p>
<pre><code class="lang-auto">print(res.shape)
</code></pre>
<blockquote>
<p>torch.Size([10, 10, 14])</p>
</blockquote>
<p>So how do I ensure that I’m using the positional embedding correctly?</p>

Flock1: <p>So I was able to solve the embedding dimension issue. For the training, I have written this:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    loss = 0
    train_model.train()
'''Returns the signal to construct and embeddings'''
    for encoded_data in tqdm(train_loader): 
        signal = signal.squeeze()

'''Since the embeddings are of different shape (128,10) or (499,10), I parse a batch of 10 embeddings'''

        for i in range(0,encoded_data.shape[1],10):

            signal_train = torch.reshape()
                
            input_data = encoded_data[:,i:i+10,:].squeeze().cuda()
            signal_train = signal_train.cuda()
            optimizer.zero_grad()
            signal_recon = train_model(input_data)

            loss = criterion(signal_recon, signal_train)
            loss.backward()
            
            optimizer.step()
            
            loss += loss.item()
</code></pre>
<p>So I wanted to know if I am doing this correctly or you recommend any changes for faster training or more accurate training?</p>