Error in Positional embedding layer

Keerthu: <p>Hello all , I’m trying to implement a GPT model .But firstly for the input part I’m trying to including positional embedding to my input embedding , but this error is occurring and I don’t know how to resolve it. Can anyone please help me out?</p>
<p>Code :<br>
from types import ModuleType<br>
class GPT(nn.Module):<br>
def <strong>init</strong>(self, n_embd, n_head, vocab_size,max_seq_len=32,device=“cpu”):<br>
super().<strong>init</strong>()<br>
self.token_embedding = nn.Embedding(vocab_size, n_embd).to(device)<br>
self.position_embedding = nn.Embedding(max_seq_len,n_embd,device=device)<br>
self.blocks = nn.Sequential(*(Block(n_embd,n_head,device) for i in range(n_head))).to(device)<br>
self.ln = nn.LayerNorm(n_embd).to(device)<br>
self.fc = nn.Linear(n_embd,64).to(device)</p>
<pre><code>def forward(self, idx, targets=None):
    B,T = idx.shape
    out = self.token_embedding(idx)
    positions = torch.arange(0, T, device=idx.device).unsqueeze(0).expand(B, -1)
    position_embedding = self.position_embedding(positions)  # (batch_size, sequence_length)
    out = out + position_embedding
    out = self.blocks(out)
    out = self.ln(out)
    logits = self.fc(out)
    print(f'Logits size{logits.size()}')
    if targets is None:
      return logits , None
    loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
    return logits,loss
</code></pre>
<p>def generate(model,idx,start_char=“F”, max_new_tokens=100, top_p=0.9, top_k=50, temperature=1):<br>
model.eval()<br>
generated = [start_char]<br>
filter_value=-float(‘Inf’)<br>
for _ in range(max_new_tokens):<br>
logits, _ = model(idx)<br>
next_char_id = torch.argmax(logits).item()<br>
next_char = [k for k, v in c2id.items() if v == next_char_id][0]<br>
current_char = next_char<br>
generated += current_char<br>
generated.append(next_token.item())<br>
return generated</p>
<h1><a name="example-usage-1" class="anchor" href="#example-usage-1"></a>Example usage</h1>
<p>vocab_size = 5000  # Example vocab size<br>
embed_size = 64  # Example embedding size<br>
max_length = 32  # Example max length</p>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
model = GPT(embed_size, 4,vocab_size,device)</p>
<p>idx = idx</p>
<p>targets = idy<br>
logits, loss = model.forward(idx,targets)</p>
<p>print(f’Logits: {logits.size()}, Loss: {loss.item()}')</p>
<p><span class="hashtag-raw">#Generating</span> word before training<br>
generated_word = generate(model,idx)<br>
print(f’Generated word before training: {generated_word}')</p>
<p>Error :<br>
TypeError                                 Traceback (most recent call last)<br>
 in &lt;cell line: 47&gt;()<br>
45<br>
46 device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
—&gt; 47 model = GPT(embed_size, 4,vocab_size,device)<br>
48<br>
49 idx = idx</p>
<p>1 frames<br>
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/sparse.py in <strong>init</strong>(self, num_embeddings, embedding_dim, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse, _weight, _freeze, device, dtype)<br>
141         self.scale_grad_by_freq = scale_grad_by_freq<br>
142         if _weight is None:<br>
 → 143             self.weight = Parameter(torch.empty((num_embeddings, embedding_dim), **factory_kwargs),<br>
144                                     requires_grad=not _freeze)<br>
145             self.reset_parameters()</p>
<p>TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=NoneType, device=str), but expected one of:</p>
<ul>
<li>(tuple of ints size, *, tuple of names names, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)</li>
<li>(tuple of ints size, *, torch.memory_format memory_format, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)</li>
</ul>

ptrblck: <p>Could you print all inputs to this line of code, please?</p>
<pre data-code-wrap="python"><code class="lang-python"> self.weight = Parameter(torch.empty((num_embeddings, embedding_dim), **factory_kwargs),
</code></pre>

Keerthu: <p>Hello, sure .<br>
num_embeddings = 64</p>
<p>I’m a bit confused on  embedding_dim value as I didn’t give that in code . Id actually related to another variable name ?</p>

ptrblck: <p>Your model’s <code>__init__</code> is most likely defining it, so try to isolate the arguments to understand why the line of code fails.</p>