Runtimer: expected scalar type Double but found Float

Kunal_Jangra: <p>I am working on sentiment analysis using GNN, While running the below code i am getting the run time error.<br>
import pandas as pd<br>
import torch<br>
from sklearn.model_selection import train_test_split</p>
<h1>
<a name="load-the-preprocessed-tweets-into-a-pandas-dataframe-1" class="anchor" href="#load-the-preprocessed-tweets-into-a-pandas-dataframe-1"></a>Load the preprocessed tweets into a pandas dataframe</h1>
<h1>
<a name="split-the-data-into-training-validation-and-testing-sets-2" class="anchor" href="#split-the-data-into-training-validation-and-testing-sets-2"></a>Split the data into training, validation, and testing sets</h1>
<p>train_data, test_data = train_test_split(df_tweets, test_size=0.2, random_state=42)<br>
train_data, val_data = train_test_split(train_data, test_size=0.2, random_state=42)</p>
<h2>
<a name="convert-the-data-to-pytorch-tensors-3" class="anchor" href="#convert-the-data-to-pytorch-tensors-3"></a>Convert the data to PyTorch tensors</h2>
<p>train_tweets = torch.DoubleTensor(train_data[‘Numerical Representations’].tolist())<br>
train_labels = torch.DoubleTensor(train_data[‘Label’].tolist())</p>
<p>val_tweets = torch.DoubleTensor(val_data[‘Numerical Representations’].tolist())<br>
val_labels = torch.DoubleTensor(val_data[‘Label’].tolist())</p>
<p>test_tweets = torch.DoubleTensor(test_data[‘Numerical Representations’].tolist())<br>
test_labels = torch.DoubleTensor(test_data[‘Label’].tolist())</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class GNNGRUModel(nn.Module):<br>
def <strong>init</strong>(self, embedding_matrix, hidden_size, num_classes):<br>
super(GNNGRUModel, self).<strong>init</strong>()</p>
<pre><code>    # Set the embedding layer with pre-trained word embeddings
    self.embedding_layer = nn.Embedding.from_pretrained(embedding_matrix, freeze=True)

    # Set the GRU layer with hidden size
    self.gru_layer = nn.GRU(embedding_matrix.shape[1], hidden_size, batch_first=True)

    # Set the fully connected layer with output size equal to the number of classes
    self.fc_layer = nn.Linear(hidden_size, num_classes)

def forward(self, inputs):
    # Extract the padded sequence tensor from inputs
    padded_sequences = inputs

    # Pass the padded sequence tensor to the embedding layer
    embedded_sequences = self.embedding_layer(padded_sequences.long())

    # Convert the embedded sequence tensor to double type
    #embedded_sequences = embedded_sequences.double()

    # Pass the embedded sequence tensor to the GRU layer
    gru_output, _ = self.gru_layer(embedded_sequences)

    # Take the last hidden state from the GRU layer as the sentence representation
    sentence_representation = gru_output[:, -1, :]

    # Pass the sentence representation tensor to the fully connected layer
    logits = self.fc_layer(sentence_representation)

    # Apply softmax activation to get the predicted probabilities
    output = F.softmax(logits, dim=1)

    return output
</code></pre>
<p>import torch</p>
<h1>
<a name="convert-the-embedding-matrix-to-a-pytorch-tensor-4" class="anchor" href="#convert-the-embedding-matrix-to-a-pytorch-tensor-4"></a>Convert the embedding matrix to a PyTorch tensor</h1>
<p>embedding_tensor = torch.from_numpy(embedding_matrix)</p>
<h1>
<a name="create-an-instance-of-the-gnngrumodel-class-5" class="anchor" href="#create-an-instance-of-the-gnngrumodel-class-5"></a>Create an instance of the GNNGRUModel class</h1>
<p>model = GNNGRUModel(embedding_tensor, hidden_size=128, num_classes=3)</p>
<h1>
<a name="move-the-model-to-the-gpu-if-available-6" class="anchor" href="#move-the-model-to-the-gpu-if-available-6"></a>Move the model to the GPU if available</h1>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
model.to(device)</p>
<h1>
<a name="set-the-loss-function-and-optimizer-7" class="anchor" href="#set-the-loss-function-and-optimizer-7"></a>Set the loss function and optimizer</h1>
<p>loss_fn = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)</p>
<p>from tqdm import tqdm</p>
<h1>
<a name="set-the-number-of-epochs-to-train-for-8" class="anchor" href="#set-the-number-of-epochs-to-train-for-8"></a>Set the number of epochs to train for</h1>
<p>num_epochs = 10</p>
<h1>
<a name="set-the-batch-size-for-training-9" class="anchor" href="#set-the-batch-size-for-training-9"></a>Set the batch size for training</h1>
<p>batch_size = 32</p>
<h1>
<a name="create-a-dataloader-for-the-training-set-10" class="anchor" href="#create-a-dataloader-for-the-training-set-10"></a>Create a DataLoader for the training set</h1>
<p>train_dataset = torch.utils.data.TensorDataset(train_tweets, train_labels)<br>
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)</p>
<h1>
<a name="create-a-dataloader-for-the-validation-set-11" class="anchor" href="#create-a-dataloader-for-the-validation-set-11"></a>Create a DataLoader for the validation set</h1>
<p>val_dataset = torch.utils.data.TensorDataset(val_tweets, val_labels)<br>
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=batch_size, shuffle=False)</p>
<h1>
<a name="train-the-model-12" class="anchor" href="#train-the-model-12"></a>Train the model</h1>
<p>for epoch in range(num_epochs):<br>
# Set the model to training mode<br>
model.train()</p>
<pre><code># Initialize the total loss for this epoch
total_loss = 0

# Loop over the batches in the training set
for batch_tweets, batch_labels in tqdm(train_loader, desc=f'Training epoch {epoch + 1}/{num_epochs}'):
    # Move the batch to the GPU if available
    batch_tweets = batch_tweets.double()
    batch_tweets = batch_tweets.to(device)
    batch_labels = batch_labels.double()
    batch_labels = batch_labels.to(device)

    # Zero out the gradients
    optimizer.zero_grad()

    # Forward pass
    batch_logits = model(batch_tweets)

    # Compute the loss
    batch_loss = loss_fn(batch_logits, batch_label)

    # Backward pass
    batch_loss.backward()

    # Update the parameters
    optimizer.step()

    # Add the batch loss to the total loss for this epoch
    total_loss += batch_loss.item()

# Compute the average loss for this epoch
avg_loss = total_loss / len(train_loader)

# Set the model to evaluation mode
model.eval()

# Initialize the number of correct predictions for this epoch
num_correct = 0

# Loop over the batches in the validation set
for batch_tweets, batch_labels in tqdm(val_loader, desc=f'Validation epoch {epoch + 1}/{num_epochs}'):
    # Move the batch to the GPU if available
    batch_tweets = batch_tweets.to(device)
    batch_labels = batch_labels.to(device)

    # Forward pass
    batch_logits = model(batch_tweets)

    # Get the predicted labels
    _, batch_preds = batch_logits.max(dim=1)

    # Compute the number of correct predictions
    num_correct += (batch_preds == batch_labels).sum().item()

# Compute the accuracy for this epoch
accuracy = num_correct / len(val_dataset)

# Print the average loss and accuracy for this epoch
print(f'Epoch {epoch + 1}/{num_epochs}, '
      f'Average Loss: {avg_loss:.4f}, '
      f'Validation Accuracy: {accuracy:.4f}')
</code></pre>

ptrblck: <p>In your code you are explicitly creating the data as <code>DoubleTensor</code>s and are also casting again to <code>double</code> inside the <code>DataLoader</code> loop.<br>
PyTorch uses <code>float32</code> by default for all parameters of the model.<br>
If you really want top use <code>float64</code>, you could cast the model’s parameters and buffers via <code>model.to(torch.float64)</code>, but note that this could have a large performance impact especially on the GPU.<br>
Assuming you don’t need this high precision you could cast the inputs to <code>float32</code>.</p>

Kunal_Jangra: <p>Thank you nut the error is still there.</p>

ptrblck: <p>Could you post the stacktrace raising this error, please?</p>

Kunal_Jangra: <hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
~\AppData\Local\Temp\ipykernel_17800\2285477610.py in <br>
117<br>
118         # Forward pass<br>
 → 119         batch_logits = model(batch_tweets)<br>
120<br>
121         # Compute the loss</p>
<p>~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1050                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1051             return forward_call(*input, **kwargs)<br>
1052         # Do not call functions when jit is used<br>
1053         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>~\AppData\Local\Temp\ipykernel_17800\2285477610.py in forward(self, inputs)<br>
49<br>
50         # Pass the embedded sequence tensor to the GRU layer<br>
—&gt; 51         gru_output, _ = self.gru_layer(embedded_sequences)<br>
52<br>
53         # Take the last hidden state from the GRU layer as the sentence representation</p>
<p>~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1050                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1051             return forward_call(*input, **kwargs)<br>
1052         # Do not call functions when jit is used<br>
1053         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>~\anaconda3\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)<br>
835         self.check_forward_args(input, hx, batch_sizes)<br>
836         if batch_sizes is None:<br>
 → 837             result = _VF.gru(input, hx, self._flat_weights, self.bias, self.num_layers,<br>
838                              self.dropout, self.training, self.bidirectional, self.batch_first)<br>
839         else:</p>
<p>RuntimeError: expected scalar type Double but found Float</p>

ptrblck: <p>Thanks for the stacktrace (you can post code snippet by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ).<br>
Are you casting the activations back to <code>double</code> before passing them to the <code>gru_layer</code>?<br>
I cannot reproduce the issue using your code:</p>
<pre><code class="lang-python">class GNNGRUModel(nn.Module):
    def __init__(self, embedding_matrix, hidden_size, num_classes):
        super(GNNGRUModel, self).__init__()
    
        # Set the embedding layer with pre-trained word embeddings
        self.embedding_layer = nn.Embedding.from_pretrained(embedding_matrix, freeze=True)
    
        # Set the GRU layer with hidden size
        self.gru_layer = nn.GRU(embedding_matrix.shape[1], hidden_size, batch_first=True)
    
        # Set the fully connected layer with output size equal to the number of classes
        self.fc_layer = nn.Linear(hidden_size, num_classes)
    
    def forward(self, inputs):
        # Extract the padded sequence tensor from inputs
        padded_sequences = inputs
    
        # Pass the padded sequence tensor to the embedding layer
        embedded_sequences = self.embedding_layer(padded_sequences.long())
    
        # Convert the embedded sequence tensor to double type
        #embedded_sequences = embedded_sequences.double()
    
        # Pass the embedded sequence tensor to the GRU layer
        gru_output, _ = self.gru_layer(embedded_sequences)
        return gru_output
        
model = GNNGRUModel(torch.randn(10, 10), 10, 10)
x = torch.randint(0, 10, (10, 10))
out = model(x)
print(out.shape)
# torch.Size([10, 10, 10])
</code></pre>

Kunal_Jangra: <p>sorry I didn’t understand. Could you please explain where i am going wrong?</p>