RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)

username: sashamyronyuk1000
cooked: <p>import sqlite3<br>
import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.utils.data as data<br>
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence<br>
import nltk<br>
from nltk.tokenize import word_tokenize<br>
from collections import Counter<br>
from sklearn.model_selection import train_test_split<br>
from nltk.translate.bleu_score import sentence_bleu<br>
import numpy as np<br>
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)</p>
<h1><a name="h-1-1" class="anchor" href="#h-1-1"></a>Етап 1: Підготовка даних</h1>
<h1><a name="sqlite-2" class="anchor" href="#sqlite-2"></a>Зчитуємо дані з бази даних і створюємо SQLite базу даних</h1>
<p>conn = sqlite3.connect(‘translations1.db’)<br>
c = conn.cursor()</p>
<h1><a name="h-3" class="anchor" href="#h-3"></a>Створення таблиці для даних</h1>
<p>c.execute(‘’‘CREATE TABLE IF NOT EXISTS translations<br>
(source_text TEXT, target_text TEXT)’‘’)</p>
<h1><a name="h-4" class="anchor" href="#h-4"></a>Додавання прикладів даних до таблиці</h1>
<p>examples = [(‘Hello world’, ‘Привіт, світе’),<br>
(‘How are you?’, ‘Як справи?’),<br>
(‘I love programming’, ‘Я люблю програмувати’),<br>
(‘Good morning’, ‘Доброго ранку’),<br>
(‘Thank you’, ‘Дякую’),<br>
(‘Where is the nearest restaurant?’, ‘Де найближчий ресторан?’),<br>
(‘What time is it?’, ‘Котра година?’),<br>
(‘I need help’, ‘Мені потрібна допомога’),<br>
(‘The weather is nice today’, ‘Сьогодні гарна погода’)]</p>
<p>c.executemany(“INSERT INTO translations (source_text, target_text) VALUES (?, ?)”, examples)</p>
<h1><a name="h-5" class="anchor" href="#h-5"></a>Зберігаємо зміни в базі даних</h1>
<p>conn.commit()</p>
<h1><a name="h-2-6" class="anchor" href="#h-2-6"></a>Етап 2: Токенізація та чисельне представлення тексту</h1>
<h1><a name="h-7" class="anchor" href="#h-7"></a>Токенізація тексту</h1>
<p>nltk.download(‘punkt’)<br>
def tokenize(text):<br>
return word_tokenize(text.lower())</p>
<h1><a name="vocabularies-8" class="anchor" href="#vocabularies-8"></a>Створення словників (vocabularies) для української та англійської мов</h1>
<p>def build_vocab(texts):<br>
word_counts = Counter()<br>
for text in texts:<br>
tokens = tokenize(text)<br>
word_counts.update(tokens)<br>
vocab = {word: idx + 2 for idx, (word, _) in enumerate(word_counts.most_common())}<br>
vocab[‘’] = 0<br>
vocab[‘’] = 1<br>
return vocab</p>
<h1><a name="h-9" class="anchor" href="#h-9"></a>Отримання списку всіх текстів з бази даних</h1>
<p>c.execute(“SELECT source_text FROM translations”)<br>
source_texts = [row[0] for row in c.fetchall()]<br>
c.execute(“SELECT target_text FROM translations”)<br>
target_texts = [row[0] for row in c.fetchall()]</p>
<h1><a name="h-10" class="anchor" href="#h-10"></a>Побудова словників для української та англійської мов</h1>
<p>source_vocab = build_vocab(source_texts)<br>
target_vocab = build_vocab(target_texts)</p>
<h1><a name="h-11" class="anchor" href="#h-11"></a>Зворотні словники для перетворення чисельних ідентифікаторів у слова</h1>
<p>reverse_source_vocab = {idx: word for word, idx in source_vocab.items()}<br>
reverse_target_vocab = {idx: word for word, idx in target_vocab.items()}</p>
<h1><a name="h-3-dataloader-12" class="anchor" href="#h-3-dataloader-12"></a>Етап 3: Створення DataLoader</h1>
<h1><a name="h-13" class="anchor" href="#h-13"></a>Розділення даних на навчальний, валідаційний та тестовий набори</h1>
<p>source_train, source_temp, target_train, target_temp = train_test_split(source_texts, target_texts, test_size=0.2, random_state=42)<br>
source_val, source_test, target_val, target_test = train_test_split(source_temp, target_temp, test_size=0.5, random_state=42)</p>
<h1><a name="h-14" class="anchor" href="#h-14"></a>Клас для навчального набору даних</h1>
<p>class TranslationDataset(data.Dataset):<br>
def <strong>init</strong>(self, source_texts, target_texts, source_vocab, target_vocab):<br>
self.source_texts = source_texts<br>
self.target_texts = target_texts<br>
self.source_vocab = source_vocab<br>
self.target_vocab = target_vocab</p>
<pre><code>def __len__(self):
    return len(self.source_texts)

def __getitem__(self, idx):
    source_text = self.source_texts[idx]
    target_text = self.target_texts[idx]
    source_tokens = [self.source_vocab.get(token, self.source_vocab['&lt;UNK&gt;']) for token in tokenize(source_text)]
    target_tokens = [self.target_vocab.get(token, self.target_vocab['&lt;UNK&gt;']) for token in tokenize(target_text)]
    return source_tokens, target_tokens
</code></pre>
<h1><a name="dataloader-15" class="anchor" href="#dataloader-15"></a>Параметри для DataLoader</h1>
<p>batch_size = 64<br>
pad_idx = source_vocab[‘’]<br>
print(data)</p>
<h1><a name="h-16" class="anchor" href="#h-16"></a>Функція для пакування послідовностей</h1>
<p>def collate_fn(data):<br>
source_seqs, target_seqs = zip(*data)<br>
source_padded = torch.nn.utils.rnn.pad_sequence([torch.LongTensor(seq) for seq in source_seqs], padding_value=pad_idx, batch_first=True)<br>
target_padded = torch.nn.utils.rnn.pad_sequence([torch.LongTensor(seq) for seq in target_seqs], padding_value=pad_idx, batch_first=True)<br>
return source_padded, target_padded</p>
<h1><a name="dataloader-17" class="anchor" href="#dataloader-17"></a>Створення DataLoader</h1>
<p>train_dataset = TranslationDataset(source_train, target_train, source_vocab, target_vocab)<br>
train_loader = data.DataLoader(train_dataset, batch_size=batch_size, collate_fn=collate_fn)</p>
<h1><a name="h-4-seq2seq-18" class="anchor" href="#h-4-seq2seq-18"></a>Етап 4: Модель Seq2Seq</h1>
<h1><a name="seq2seq-19" class="anchor" href="#seq2seq-19"></a>Клас моделі Seq2Seq</h1>
<p>class Seq2Seq(nn.Module):<br>
def <strong>init</strong>(self, source_vocab_size, target_vocab_size, embed_size, hidden_size):<br>
super(Seq2Seq, self).<strong>init</strong>()<br>
self.source_embedding = nn.Embedding(source_vocab_size, embed_size)<br>
self.encoder = nn.LSTM(embed_size, hidden_size, batch_first=True)<br>
self.target_embedding = nn.Embedding(target_vocab_size, embed_size)<br>
self.decoder = nn.LSTM(embed_size, hidden_size, batch_first=True)<br>
self.output_layer = nn.Linear(hidden_size, target_vocab_size)</p>
<pre><code>def forward(self, source_seqs, target_seqs, source_lengths, target_lengths):
    source_embedded = self.source_embedding(source_seqs)
    packed_source = pack_padded_sequence(source_embedded, source_lengths, batch_first=True, enforce_sorted=False)
    encoder_outputs, (encoder_hidden, _) = self.encoder(packed_source)

    target_embedded = self.target_embedding(target_seqs)
    packed_target = pack_padded_sequence(target_embedded, target_lengths, batch_first=True, enforce_sorted=False)
    decoder_outputs, _ = self.decoder(packed_target, (encoder_hidden, _))
    unpacked_decoder_outputs, _ = pad_packed_sequence(decoder_outputs, batch_first=True)

    output_logits = self.output_layer(unpacked_decoder_outputs)
    return output_logits
</code></pre>
<h1><a name="h-20" class="anchor" href="#h-20"></a>Параметри моделі</h1>
<p>embed_size = 256<br>
hidden_size = 512</p>
<h1><a name="h-21" class="anchor" href="#h-21"></a>Створення моделі</h1>
<p>model = Seq2Seq(len(source_vocab), len(target_vocab), embed_size, hidden_size)</p>
<h1><a name="h-5-22" class="anchor" href="#h-5-22"></a>Етап 5: Навчання моделі</h1>
<h1><a name="h-23" class="anchor" href="#h-23"></a>Функція втрати та оптимізатор</h1>
<p>criterion = nn.CrossEntropyLoss(ignore_index=pad_idx)<br>
optimizer = optim.Adam(model.parameters())</p>
<h1><a name="h-24" class="anchor" href="#h-24"></a>Функція навчання</h1>
<h1><a name="h-25" class="anchor" href="#h-25"></a>Функція навчання</h1>
<h1><a name="h-26" class="anchor" href="#h-26"></a>Функція навчання</h1>
<p>def train(model, iterator, optimizer, criterion, clip):<br>
model.train()<br>
epoch_loss = 0<br>
for source_seqs, target_seqs in iterator:<br>
optimizer.zero_grad()<br>
source_lengths = torch.LongTensor([len(seq) for seq in source_seqs]).cpu()<br>
target_lengths = torch.LongTensor([len(seq) for seq in target_seqs]).cpu()<br>
source_seqs = source_seqs.to(device)<br>
target_seqs = target_seqs.to(device)</p>
<pre><code>    # Перемістимо створення source_embedded на пристрій CUDA
    source_seqs = source_seqs.long()
    target_seqs = target_seqs.long()
    source_embedded = model.source_embedding(source_seqs).to(device)
    
    output_logits = model(source_embedded, target_seqs, source_lengths, target_lengths)
    output_logits_dim = output_logits.shape[-1]
    output_logits = output_logits[:, 1:].reshape(-1, output_logits_dim)
    target_seqs = target_seqs[:, 1:].reshape(-1)
    loss = criterion(output_logits, target_seqs)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
    optimizer.step()
    epoch_loss += loss.item()

return epoch_loss / len(iterator)

# Перемістилось зайвий відступ
source_embedded = model.source_embedding(source_seqs.to(device))
target_seqs = target_seqs.to(device).long()
source_seqs = source_seqs.to(device).long()
# Перемістилось зайвий відступ
</code></pre>
<h1><a name="h-27" class="anchor" href="#h-27"></a>Навчання моделі</h1>
<p>N_EPOCHS = 10<br>
CLIP = 1</p>
<p>for epoch in range(N_EPOCHS):<br>
train_loss = train(model, train_loader, optimizer, criterion, CLIP)<br>
print(f’Epoch: {epoch+1:02}‘)<br>
print(f’\tTrain Loss: {train_loss:.3f}')</p>
<h1><a name="h-6-28" class="anchor" href="#h-6-28"></a>Етап 6: Оцінка та тестування моделі</h1>
<h1><a name="h-29" class="anchor" href="#h-29"></a>Функція для перекладу речень</h1>
<p>def translate_sentence(sentence, model, source_vocab, target_vocab, max_len=50):<br>
model.eval()<br>
tokens = tokenize(sentence)<br>
source_tokens = [source_vocab.get(token, source_vocab[‘’]) for token in tokens]<br>
source_tensor = torch.LongTensor(source_tokens).unsqueeze(0).to(device)  # Convert to torch.LongTensor and unsqueeze<br>
target_tensor = torch.LongTensor([[target_vocab[‘’]]]).to(device)  # Convert to torch.LongTensor</p>
<pre><code>with torch.no_grad():
    encoder_outputs, (encoder_hidden, _) = model.encoder(source_tensor, [len(source_tokens)])

target_token = target_vocab['&lt;SOS&gt;']
target_tokens = []
for _ in range(max_len):
    target_tensor = torch.LongTensor([[target_token]]).to(device)
    with torch.no_grad():
        decoder_outputs, (encoder_hidden, _) = model.decoder(target_tensor, (encoder_hidden, _))
    
    output_logits = model.output_layer(decoder_outputs)
    _, top_token = output_logits.squeeze(0).topk(1)
    target_token = top_token.item()
    if target_token == target_vocab['&lt;EOS&gt;']:
        break
    target_tokens.append(target_token)

translation = [reverse_target_vocab[token] for token in target_tokens]
return ' '.join(translation)
</code></pre>
<h1><a name="h-30" class="anchor" href="#h-30"></a>Переклад речення з англійської на українську</h1>
<p>source_sentence = “Hello world”<br>
translated_sentence = translate_sentence(source_sentence, model, source_vocab, target_vocab)<br>
print(f’Source: {source_sentence}‘)<br>
print(f’Translated: {translated_sentence}’)</p>
<h1><a name="h-7-31" class="anchor" href="#h-7-31"></a>Етап 7: Збереження моделі та результатів</h1>
<h1><a name="h-32" class="anchor" href="#h-32"></a>Збереження моделі</h1>
<p>torch.save(model.state_dict(), ‘seq2seq_model.pth’)</p>
<h1><a name="h-33" class="anchor" href="#h-33"></a>Оцінка моделі на тестовому наборі</h1>
<p>test_dataset = TranslationDataset(source_test, target_test, source_vocab, target_vocab)<br>
test_loader = data.DataLoader(test_dataset, batch_size=1, collate_fn=collate_fn)</p>
<p>def evaluate_bleu(model, test_loader, source_vocab, target_vocab):<br>
model.eval()<br>
bleu_scores = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for source_seqs, target_seqs, source_lengths, target_lengths in test_loader:<br>
source_sentence = ’ '.join([reverse_source_vocab[token.item()] for token in source_seqs[0]])<br>
target_sentence = ’ '.join([reverse_target_vocab[token.item()] for token in target_seqs[0]])<br>
translated_sentence = translate_sentence(source_sentence, model, source_vocab, target_vocab)<br>
bleu = sentence_bleu([target_sentence.split()], translated_sentence.split())<br>
bleu_scores.append(bleu)<br>
avg_bleu = np.mean(bleu_scores)<br>
return avg_bleu</p>
<h2><a name="avg_bleu_score-evaluate_bleumodel-test_loader-source_vocab-target_vocab-printfaverage-bleu-score-on-test-data-avg_bleu_score4f-it-throws-an-error-at-startup-nltk_data-downloading-package-punkt-to-nltk_data-cuserssashaappdataroamingnltk_data-nltk_data-package-punkt-is-already-up-to-date-module-torchutilsdata-from-cuserssashaprojeckmyvenvlibsite-packagestorchutilsdata__init__py-34" class="anchor" href="#avg_bleu_score-evaluate_bleumodel-test_loader-source_vocab-target_vocab-printfaverage-bleu-score-on-test-data-avg_bleu_score4f-it-throws-an-error-at-startup-nltk_data-downloading-package-punkt-to-nltk_data-cuserssashaappdataroamingnltk_data-nltk_data-package-punkt-is-already-up-to-date-module-torchutilsdata-from-cuserssashaprojeckmyvenvlibsite-packagestorchutilsdata__init__py-34"></a>avg_bleu_score = evaluate_bleu(model, test_loader, source_vocab, target_vocab)<br>
print(f’Average BLEU Score on Test Data: {avg_bleu_score:.4f}')<br>
It throws an error at startup [nltk_data] Downloading package punkt to<br>
[nltk_data]     C:\Users\Sasha\AppData\Roaming\nltk_data…<br>
[nltk_data]   Package punkt is already up-to-date!<br>
&lt;module ‘torch.utils.data’ from ‘C:\Users\Sasha\Projeck\myvenv\lib\site-packages\torch\utils\data\<strong>init</strong>.py’&gt;</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[18], line 195<br>
192 CLIP = 1<br>
194 for epoch in range(N_EPOCHS):<br>
 → 195     train_loss = train(model, train_loader, optimizer, criterion, CLIP)<br>
196     print(f’Epoch: {epoch+1:02}‘)<br>
197     print(f’\tTrain Loss: {train_loss:.3f}')</p>
<p>Cell In[18], line 169, in train(model, iterator, optimizer, criterion, clip)<br>
166 target_seqs = target_seqs.long()<br>
167 source_embedded = model.source_embedding(source_seqs).to(device)<br>
 → 169 output_logits = model(source_embedded, target_seqs, source_lengths, target_lengths)<br>
170 output_logits_dim = output_logits.shape[-1]<br>
171 output_logits = output_logits[:, 1:].reshape(-1, output_logits_dim)</p>
<p>File ~\Projeck\myvenv\lib\site-packages\torch\nn\modules\module.py:1518, in Module._wrapped_call_impl(self, *args, **kwargs)<br>
1516     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]<br>
1517 else:<br>
 → 1518     return self._call_impl(*args, **kwargs)</p>
<p>File ~\Projeck\myvenv\lib\site-packages\torch\nn\modules\module.py:1527, in Module._call_impl(self, *args, **kwargs)<br>
1522 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1523 # this function, and just call forward.<br>
1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1525         or _global_backward_pre_hooks or _global_backward_hooks<br>
1526         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1527     return forward_call(*args, **kwargs)<br>
1529 try:<br>
1530     result = None</p>
<p>Cell In[18], line 126, in Seq2Seq.forward(self, source_seqs, target_seqs, source_lengths, target_lengths)<br>
125 def forward(self, source_seqs, target_seqs, source_lengths, target_lengths):<br>
 → 126     source_embedded = self.source_embedding(source_seqs)<br>
127     packed_source = pack_padded_sequence(source_embedded, source_lengths, batch_first=True, enforce_sorted=False)<br>
128     encoder_outputs, (encoder_hidden, _) = self.encoder(packed_source)</p>
<p>File ~\Projeck\myvenv\lib\site-packages\torch\nn\modules\module.py:1518, in Module._wrapped_call_impl(self, *args, **kwargs)<br>
1516     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]<br>
1517 else:<br>
 → 1518     return self._call_impl(*args, **kwargs)</p>
<p>File ~\Projeck\myvenv\lib\site-packages\torch\nn\modules\module.py:1527, in Module._call_impl(self, *args, **kwargs)<br>
1522 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1523 # this function, and just call forward.<br>
1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1525         or _global_backward_pre_hooks or _global_backward_hooks<br>
1526         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1527     return forward_call(*args, **kwargs)<br>
1529 try:<br>
1530     result = None</p>
<p>File ~\Projeck\myvenv\lib\site-packages\torch\nn\modules\sparse.py:162, in Embedding.forward(self, input)<br>
161 def forward(self, input: Tensor) → Tensor:<br>
 → 162     return F.embedding(<br>
163         input, self.weight, self.padding_idx, self.max_norm,<br>
164         self.norm_type, self.scale_grad_by_freq, self.sparse)</p>
<p>File ~\Projeck\myvenv\lib\site-packages\torch\nn\functional.py:2233, in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)<br>
2227     # Note [embedding_renorm set_grad_enabled]<br>
2228     # XXX: equivalent to<br>
2229     # with torch.no_grad():<br>
2230     #   torch.embedding_renorm_<br>
2231     # remove once script supports set_grad_enabled<br>
2232     <em>no_grad_embedding_renorm</em>(weight, input, max_norm, norm_type)<br>
 → 2233 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)</p>
<p>RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘indices’ to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sashamyronyuk1000" data-post="1" data-topic="193651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sashamyronyuk1000/48/66222_2.png" class="avatar"> sashamyronyuk1000:</div>
<blockquote>
<p>return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)</p>
<p>RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘indices’ to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)</p>
</blockquote>
</aside>
<p>Embeddings require the input tensor to contain indices as long or int values while you are passing floating point values to it. Fix it and it should work.</p>