Why facing "CUDA error: device-side assert triggered" while training LSTM model?

Atharva_Mishra: <p>I am learning Pytorch and deep learning, I am working on a dataset containing 4-features. My problem statement is multiclass classification problem, total 9 possible output 1 to 9.</p>
<p>Gene which is categorical type.<br>
Variation which is categorical type.<br>
Text which is textual data.<br>
My LSTM model have 2 embedding layers for categorical data and 1 for textual data, 1 LSTM with layers=1(for testing only).</p>
<p>I have converted my textual data to numerical representation. Encoded Categorical data using LabelEncoder()</p>
<p>Using DataLoader for loading data in batch and using collate_fn() for truncating (because texts are too long) and padding on each batch.</p>
<p>As my problem statement belongs to multiclass classification, I am using torch.nn.CrossEntropyLoss(weight=class_weights) as a loss function and Adam as an optimizer.</p>
<p>As I said texts are too long so my collate_fn() function will take batch as an input and each data in batch are already converted in numerical representation and here comparing if size of each text is greater then 1500 if yes truncate them and then perform padding.</p>
<p>I have RTX3050 with 4gb of VRAM. So decided to truncate earlier it was giving cuda output of memory error in first forward pass only i.e in:</p>
<p>outputs = model(text_input.long(), gene_input.long(), variance_input.long())</p>
<p>I trained my model for only 1-epcoch training goes well(I mean no error) but during validation, I faced following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[18], line 58
     55 print(type(labels))
     57 outputs = model(text_input.long(), gene_input.long(), variance_input.long())
---&gt; 58 print(outputs)
     59 print(outputs.shape)
     60 print(type(outputs))

File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch\_tensor.py:568, in Tensor.__repr__(self, tensor_contents)
    564     return handle_torch_function(
    565         Tensor.__repr__, (self,), self, tensor_contents=tensor_contents
    566     )
    567 # All strings are unicode in Python 3.
--&gt; 568 return torch._tensor_str._str(self, tensor_contents=tensor_contents)

File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch\_tensor_str.py:704, in _str(self, tensor_contents)
    702 with torch.no_grad(), torch.utils._python_dispatch._disable_current_modes():
    703     guard = torch._C._DisableFuncTorch()
--&gt; 704     return _str_intern(self, tensor_contents=tensor_contents)

File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch\_tensor_str.py:621, in _str_intern(inp, tensor_contents)
    619                     tensor_str = _tensor_str(self.to_dense(), indent)
    620                 else:
--&gt; 621                     tensor_str = _tensor_str(self, indent)
...
    151         return

RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
</code></pre>
<p>As we can see in code during print(outputs) I am getting error this is not the case in validation period I faced this error to early or after completing some% of validation, but only statements having outputs variable.</p>
<p>I am sharing my Model and Training code as bellow:</p>
<p><strong>MODEL:</strong></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

class MultiClassLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes, gene_size, variance_size, gene_emb_dim, variance_emb_dim):
        super(MultiClassLSTM, self).__init__()

        # Text feature embedding + LSTM
        self.text_embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(input_size=embed_dim, hidden_size=hidden_dim, num_layers=1, batch_first=True)

        # Categorical feature embeddings
        self.gene_embedding = nn.Embedding(gene_size, gene_emb_dim)
        self.variance_embedding = nn.Embedding(variance_size, variance_emb_dim)
        # Fully connected layer for classification
        self.fc = nn.Sequential(
            nn.Linear(hidden_dim + gene_emb_dim + variance_emb_dim, 128),
            nn.ReLU(),
            nn.Linear(128, num_classes)
        )

    def forward(self, text_input, gene_input, variance_input):
        # Process text input through embedding and LSTM
        text_embedded = self.text_embedding(text_input)
        lstm_out, _ = self.lstm(text_embedded)
        lstm_out = lstm_out[:, -1, :]  # Take the last hidden state

        # Process categorical inputs through embeddings
        gene_embedded = self.gene_embedding(gene_input).squeeze(1)
        variance_embedded = self.variance_embedding(variance_input).squeeze(1)

        # Concatenate all features
        combined = torch.cat((lstm_out, gene_embedded, variance_embedded), dim=1)

        # Classification output
        output = self.fc(combined)
        return output


# Model Initialization
model = MultiClassLSTM(vocab_size, embed_dim, hidden_dim, num_classes, gene_size, variance_size, gene_emb_dim, variance_emb_dim)


y_full_np = np.concatenate([y_train, y_test, y_val])  # Full dataset labels
# unique_classes = np.unique(y_full_np)[1:]
unique_classes = np.array([0,1,2,3,4,5,6,7,8])
# print(unique_classes)
class_weights = compute_class_weight(class_weight="balanced", classes=np.array([0,1,2,3,4,5,6,7,8]), y=y_full_np)
class_weights = torch.tensor(class_weights, dtype=torch.float32, device=device)

# Define loss function with class weights
criterion = torch.nn.CrossEntropyLoss(weight=class_weights)

optimizer = optim.Adam(model.parameters(), lr=0.001)

optimizer.zero_grad()
</code></pre>
<p><strong>TRANING</strong></p>
<pre><code class="lang-auto">num_epochs = 1
train_losses = []
val_losses = []
os.environ["TORCH_USE_CUDA_DSA"] = "1"
import os
# os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:2024"

model.to(device)
for epoch in range(num_epochs):
    # torch.cuda.empty_cache()
    model.train()  # Set model to training mode
    total_train_loss = 0

    for batch in tqdm(train_dataloader, desc=f"Epoch {epoch+1}/{num_epochs} [Training]"):
        text_input, gene_input, variance_input, labels = batch

        # Move to device (if using GPU)
        text_input = text_input.to(device)
        gene_input = gene_input.to(device)
        variance_input = variance_input.to(device)
        labels = labels.to(device)  # Labels should be integer class indices

        # print(text_input.device, gene_input.device, variance_input.device, labels.device)

        optimizer.zero_grad()  # Clear previous gradients
        
        outputs = model(text_input.long(), gene_input.long(), variance_input.long())

        # Compute Log Loss
        loss = criterion(outputs, labels)

        # Backward pass
        loss.backward()
        optimizer.step()

        total_train_loss += loss.item()

    # Compute average training loss
    avg_train_loss = total_train_loss / len(train_dataloader)
    train_losses.append(avg_train_loss)

    # ================== Validation Phase ==================
    model.eval()  # Set model to evaluation mode
    total_val_loss = []

    with torch.no_grad():  # No gradient calculation during validation
        for batch in tqdm(validation_dataloader, desc=f"Epoch {epoch+1}/{num_epochs} [Validation]"):
            text_input, gene_input, variance_input, labels = batch
            text_input = text_input.to(device)
            gene_input = gene_input.to(device)
            variance_input = variance_input.to(device)
            labels = labels.to(device)
            print(labels)
            print(labels.shape)
            print(type(labels))

            outputs = model(text_input.long(), gene_input.long(), variance_input.long())
            print(outputs)
            print(outputs.shape)
            print(type(outputs))
            loss = criterion(outputs, labels)
            print(loss)          
            total_val_loss.append(loss.item())
            gc.collect()
            torch.cuda.empty_cache()
            print("----------------")

    avg_val_loss = sum(total_val_loss) / len(validation_dataloader)
    val_losses.append(avg_val_loss)

    print(f"Epoch [{epoch+1}/{num_epochs}], Train Loss: {avg_train_loss:.4f}, Val Loss: {avg_val_loss:.4f}")

# Store losses for future use
torch.save({'train_loss': train_losses, 'val_loss': val_losses}, 'losses.pth')
</code></pre>
<p>I used some print statement to see if shape or datatype is creating problem, I have deleted the code, but I tested if in output I am getting nan or inf because of learning rate but didn’t help. I saw some similar problem on pytorch-forum as well but didn’t understand.</p>
<p>Thanks in advance. I hope to hear from you soon.</p>

ptrblck: <aside class="quote no-group" data-username="Atharva_Mishra" data-post="1" data-topic="216698">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atharva_mishra/48/75666_2.png" class="avatar"> Atharva_Mishra:</div>
<blockquote>
<p>total 9 possible output 1 to 9.</p>
</blockquote>
</aside>
<p>Note that PyTorch often expects indices starting from 0, e.g. in <code>nn.CrossEntropyLoss</code> the target for 9 classes should contain indices in <code>[0, 8]</code>. Passing an index outside of this valid range will cause the device assert errors.</p>

Atharva_Mishra: <p>ok, let me see, but what i have saw or noticed this is not the case may be I am wrong, let me see Thanks for you response.</p>

Atharva_Mishra: <h2><a name="p-465594-epoch-11-training-100-530530-00050000-9433its-epoch-11-validation-1-1132-00000021-620itstensor6-0-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01172-01543-05504-02245-00286-00923-03298-21161-18404-01884-09037-04673-00721-11703-06113-07804-28961-22404-03921-00559-13258-04566-02808-01708-01426-14904-16001-05576-02592-00398-03589-04208-00898-08281-21817-21289-devicecuda0-torchsize4-9-class-torchtensor-tensor20431-devicecuda0-1" class="anchor" href="#p-465594-epoch-11-training-100-530530-00050000-9433its-epoch-11-validation-1-1132-00000021-620itstensor6-0-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01172-01543-05504-02245-00286-00923-03298-21161-18404-01884-09037-04673-00721-11703-06113-07804-28961-22404-03921-00559-13258-04566-02808-01708-01426-14904-16001-05576-02592-00398-03589-04208-00898-08281-21817-21289-devicecuda0-torchsize4-9-class-torchtensor-tensor20431-devicecuda0-1"></a>Epoch 1/1 [Training]: 100%|██████████| 530/530 [00:05&lt;00:00, 94.33it/s]<br>
Epoch 1/1 [Validation]:   1%|          | 1/132 [00:00&lt;00:21,  6.20it/s]tensor([6, 0, 5, 2], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.1172,  0.1543, -0.5504,  0.2245, -0.0286,  0.0923,  0.3298, -2.1161,<br>
-1.8404],<br>
[ 0.1884, -0.9037, -0.4673, -0.0721,  1.1703,  0.6113, -0.7804, -2.8961,<br>
-2.2404],<br>
[ 0.3921, -0.0559, -1.3258,  0.4566,  0.2808, -0.1708, -0.1426, -1.4904,<br>
-1.6001],<br>
[-0.5576, -0.2592, -0.0398, -0.3589, -0.4208,  0.0898,  0.8281, -2.1817,<br>
-2.1289]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.0431, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor0-6-6-5-devicecuda0-torchsize4-class-torchtensor-tensor-03324-11683-09862-04847-17634-04665-15038-38269-27187-03151-08455-07605-01490-08000-09488-13387-14526-18561-02988-02303-00565-10323-02438-09100-03279-20378-25445-05214-11528-06574-00817-12476-07205-10407-33927-25857-devicecuda0-torchsize4-9-class-torchtensor-tensor16296-devicecuda0-epoch-11-validation-2-3132-00000016-787its-tensor5-6-3-0-devicecuda0-torchsize4-class-torchtensor-tensor-02032-04361-06183-18954-07922-14702-01195-24953-26830-02581-05257-10674-00423-01433-02777-03005-31134-22096-00335-02139-05577-03850-01559-05312-00724-08430-08123-01569-02349-06334-09193-02774-06634-04086-19756-10669-devicecuda0-torchsize4-9-class-torchtensor-tensor27672-devicecuda0-2" class="anchor" href="#p-465594-tensor0-6-6-5-devicecuda0-torchsize4-class-torchtensor-tensor-03324-11683-09862-04847-17634-04665-15038-38269-27187-03151-08455-07605-01490-08000-09488-13387-14526-18561-02988-02303-00565-10323-02438-09100-03279-20378-25445-05214-11528-06574-00817-12476-07205-10407-33927-25857-devicecuda0-torchsize4-9-class-torchtensor-tensor16296-devicecuda0-epoch-11-validation-2-3132-00000016-787its-tensor5-6-3-0-devicecuda0-torchsize4-class-torchtensor-tensor-02032-04361-06183-18954-07922-14702-01195-24953-26830-02581-05257-10674-00423-01433-02777-03005-31134-22096-00335-02139-05577-03850-01559-05312-00724-08430-08123-01569-02349-06334-09193-02774-06634-04086-19756-10669-devicecuda0-torchsize4-9-class-torchtensor-tensor27672-devicecuda0-2"></a>tensor([0, 6, 6, 5], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.3324, -1.1683, -0.9862,  0.4847,  1.7634,  0.4665, -1.5038, -3.8269,<br>
-2.7187],<br>
[-0.3151,  0.8455, -0.7605,  0.1490, -0.8000, -0.9488,  1.3387, -1.4526,<br>
-1.8561],<br>
[-0.2988,  0.2303,  0.0565,  1.0323, -0.2438, -0.9100,  0.3279, -2.0378,<br>
-2.5445],<br>
[ 0.5214, -1.1528, -0.6574,  0.0817,  1.2476,  0.7205, -1.0407, -3.3927,<br>
-2.5857]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.6296, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   2%|▏         | 3/132 [00:00&lt;00:16,  7.87it/s]----------------<br>
tensor([5, 6, 3, 0], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.2032,  0.4361, -0.6183,  1.8954, -0.7922, -1.4702, -0.1195, -2.4953,<br>
-2.6830],<br>
[ 0.2581, -0.5257, -1.0674,  0.0423,  0.1433,  0.2777,  0.3005, -3.1134,<br>
-2.2096],<br>
[ 0.0335,  0.2139, -0.5577,  0.3850, -0.1559, -0.5312,  0.0724, -0.8430,<br>
-0.8123],<br>
[ 0.1569, -0.2349, -0.6334, -0.9193,  0.2774,  0.6634,  0.4086, -1.9756,<br>
-1.0669]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.7672, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor5-4-3-3-devicecuda0-torchsize4-class-torchtensor-tensor-07228-13446-09910-01117-12572-12824-12669-39945-28691-01884-09037-04673-00721-11703-06113-07804-28961-22404-04943-09349-08567-02696-10498-05423-07203-25375-18992-04857-02578-10885-15036-04540-06005-09201-32721-27364-devicecuda0-torchsize4-9-class-torchtensor-tensor12083-devicecuda0-epoch-11-validation-4-5132-00000016-776its-tensor1-6-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01827-04180-10472-01140-06313-00492-08390-25656-17660-04286-08027-04469-14552-05709-08960-03469-32073-28408-02910-09920-10511-06854-04868-25009-02337-42157-33742-01958-04928-00435-00273-00136-00224-01098-21593-19146-devicecuda0-torchsize4-9-class-torchtensor-tensor15772-devicecuda0-3" class="anchor" href="#p-465594-tensor5-4-3-3-devicecuda0-torchsize4-class-torchtensor-tensor-07228-13446-09910-01117-12572-12824-12669-39945-28691-01884-09037-04673-00721-11703-06113-07804-28961-22404-04943-09349-08567-02696-10498-05423-07203-25375-18992-04857-02578-10885-15036-04540-06005-09201-32721-27364-devicecuda0-torchsize4-9-class-torchtensor-tensor12083-devicecuda0-epoch-11-validation-4-5132-00000016-776its-tensor1-6-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01827-04180-10472-01140-06313-00492-08390-25656-17660-04286-08027-04469-14552-05709-08960-03469-32073-28408-02910-09920-10511-06854-04868-25009-02337-42157-33742-01958-04928-00435-00273-00136-00224-01098-21593-19146-devicecuda0-torchsize4-9-class-torchtensor-tensor15772-devicecuda0-3"></a>tensor([5, 4, 3, 3], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.7228, -1.3446, -0.9910, -0.1117,  1.2572,  1.2824, -1.2669, -3.9945,<br>
-2.8691],<br>
[ 0.1884, -0.9037, -0.4673, -0.0721,  1.1703,  0.6113, -0.7804, -2.8961,<br>
-2.2404],<br>
[ 0.4943, -0.9349, -0.8567, -0.2696,  1.0498,  0.5423, -0.7203, -2.5375,<br>
-1.8992],<br>
[ 0.4857, -0.2578, -1.0885,  1.5036,  0.4540, -0.6005, -0.9201, -3.2721,<br>
-2.7364]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.2083, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   4%|▍         | 5/132 [00:00&lt;00:16,  7.76it/s]----------------<br>
tensor([1, 6, 5, 2], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.1827,  0.4180, -1.0472, -0.1140, -0.6313, -0.0492,  0.8390, -2.5656,<br>
-1.7660],<br>
[ 0.4286, -0.8027, -0.4469,  1.4552, -0.5709, -0.8960, -0.3469, -3.2073,<br>
-2.8408],<br>
[-0.2910, -0.9920, -1.0511, -0.6854,  0.4868,  2.5009, -0.2337, -4.2157,<br>
-3.3742],<br>
[-0.1958, -0.4928,  0.0435, -0.0273,  0.0136,  0.0224,  0.1098, -2.1593,<br>
-1.9146]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.5772, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor6-5-6-3-devicecuda0-torchsize4-class-torchtensor-tensor-02412-06586-09805-08835-05440-09555-01473-24310-15796-02910-09920-10511-06854-04868-25009-02337-42157-33742-02032-04361-06183-18954-07922-14702-01195-24953-26830-11571-08971-11486-17376-00474-05935-13113-37319-32834-devicecuda0-torchsize4-9-class-torchtensor-tensor09657-devicecuda0-epoch-11-validation-5-7132-00000015-814its-tensor0-2-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-11050-10732-13967-16984-01679-01173-12775-47368-40365-03627-00192-04877-08786-00692-06484-00771-17631-22243-04638-06656-06289-17969-01436-07338-07436-36590-32770-01481-07261-11944-01191-02490-05490-06908-16781-17578-devicecuda0-torchsize4-9-class-torchtensor-tensor21222-devicecuda0-4" class="anchor" href="#p-465594-tensor6-5-6-3-devicecuda0-torchsize4-class-torchtensor-tensor-02412-06586-09805-08835-05440-09555-01473-24310-15796-02910-09920-10511-06854-04868-25009-02337-42157-33742-02032-04361-06183-18954-07922-14702-01195-24953-26830-11571-08971-11486-17376-00474-05935-13113-37319-32834-devicecuda0-torchsize4-9-class-torchtensor-tensor09657-devicecuda0-epoch-11-validation-5-7132-00000015-814its-tensor0-2-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-11050-10732-13967-16984-01679-01173-12775-47368-40365-03627-00192-04877-08786-00692-06484-00771-17631-22243-04638-06656-06289-17969-01436-07338-07436-36590-32770-01481-07261-11944-01191-02490-05490-06908-16781-17578-devicecuda0-torchsize4-9-class-torchtensor-tensor21222-devicecuda0-4"></a>tensor([6, 5, 6, 3], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.2412, -0.6586, -0.9805, -0.8835,  0.5440,  0.9555,  0.1473, -2.4310,<br>
-1.5796],<br>
[-0.2910, -0.9920, -1.0511, -0.6854,  0.4868,  2.5009, -0.2337, -4.2157,<br>
-3.3742],<br>
[ 0.2032,  0.4361, -0.6183,  1.8954, -0.7922, -1.4702, -0.1195, -2.4953,<br>
-2.6830],<br>
[ 1.1571, -0.8971, -1.1486,  1.7376,  0.0474, -0.5935, -1.3113, -3.7319,<br>
-3.2834]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(0.9657, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   5%|▌         | 7/132 [00:00&lt;00:15,  8.14it/s]----------------<br>
tensor([0, 2, 3, 1], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 1.1050, -1.0732, -1.3967,  1.6984,  0.1679, -0.1173, -1.2775, -4.7368,<br>
-4.0365],<br>
[ 0.3627, -0.0192, -0.4877,  0.8786, -0.0692, -0.6484,  0.0771, -1.7631,<br>
-2.2243],<br>
[ 0.4638, -0.6656, -0.6289,  1.7969, -0.1436, -0.7338, -0.7436, -3.6590,<br>
-3.2770],<br>
[-0.1481,  0.7261, -1.1944,  0.1191, -0.2490, -0.5490,  0.6908, -1.6781,<br>
-1.7578]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.1222, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor1-5-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-02084-03109-05869-03480-02416-00612-03557-07210-05134-00914-04602-11326-16102-07496-14515-02776-35171-20348-02383-02433-07864-13938-02066-05387-00077-24466-29635-01002-00886-10583-03418-03617-02141-05558-15194-12726-devicecuda0-torchsize4-9-class-torchtensor-tensor12938-devicecuda0-epoch-11-validation-7-9132-00010014-832its-tensor1-0-0-0-devicecuda0-torchsize4-class-torchtensor-tensor-00827-09098-15061-00083-05200-08133-12496-13435-15232-00197-00608-01855-03570-01213-07042-00125-14624-12429-12857-13768-15441-18789-01597-00293-17447-50522-41930-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor17246-devicecuda0-5" class="anchor" href="#p-465594-tensor1-5-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-02084-03109-05869-03480-02416-00612-03557-07210-05134-00914-04602-11326-16102-07496-14515-02776-35171-20348-02383-02433-07864-13938-02066-05387-00077-24466-29635-01002-00886-10583-03418-03617-02141-05558-15194-12726-devicecuda0-torchsize4-9-class-torchtensor-tensor12938-devicecuda0-epoch-11-validation-7-9132-00010014-832its-tensor1-0-0-0-devicecuda0-torchsize4-class-torchtensor-tensor-00827-09098-15061-00083-05200-08133-12496-13435-15232-00197-00608-01855-03570-01213-07042-00125-14624-12429-12857-13768-15441-18789-01597-00293-17447-50522-41930-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor17246-devicecuda0-5"></a>tensor([1, 5, 3, 1], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.2084,  0.3109, -0.5869, -0.3480, -0.2416, -0.0612,  0.3557, -0.7210,<br>
-0.5134],<br>
[-0.0914, -0.4602, -1.1326, -1.6102,  0.7496,  1.4515,  0.2776, -3.5171,<br>
-2.0348],<br>
[ 0.2383, -0.2433, -0.7864,  1.3938, -0.2066, -0.5387, -0.0077, -2.4466,<br>
-2.9635],<br>
[ 0.1002,  0.0886, -1.0583, -0.3418, -0.3617,  0.2141,  0.5558, -1.5194,<br>
-1.2726]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.2938, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   7%|▋         | 9/132 [00:01&lt;00:14,  8.32it/s]----------------<br>
tensor([1, 0, 0, 0], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.0827,  0.9098, -1.5061,  0.0083, -0.5200, -0.8133,  1.2496, -1.3435,<br>
-1.5232],<br>
[-0.0197, -0.0608, -0.1855,  0.3570, -0.1213, -0.7042,  0.0125, -1.4624,<br>
-1.2429],<br>
[ 1.2857, -1.3768, -1.5441,  1.8789,  0.1597, -0.0293, -1.7447, -5.0522,<br>
-4.1930],<br>
[ 0.1521,  0.1529, -0.6599,  2.0708, -0.4169, -1.6593, -0.1911, -2.6210,<br>
-3.0652]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.7246, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor6-3-0-6-devicecuda0-torchsize4-class-torchtensor-tensor-00912-00718-11203-05582-02274-02479-01236-20472-15937-01362-04223-09636-03833-02258-03686-07650-09438-07804-01861-01010-06031-00587-00172-05022-04774-15771-15994-01907-00961-14858-01823-00920-04004-00362-18670-15711-devicecuda0-torchsize4-9-class-torchtensor-tensor21856-devicecuda0-epoch-11-validation-8-11132-00010014-845its-tensor6-1-6-6-devicecuda0-torchsize4-class-torchtensor-tensor-00159-03259-10007-01159-04197-09344-07106-19948-14186-05346-09564-02370-09762-07139-10262-07824-18359-23684-00397-04967-00271-02689-02430-02428-02138-22142-19273-02429-01047-05400-07540-01807-05147-02106-14485-17857-devicecuda0-torchsize4-9-class-torchtensor-tensor15966-devicecuda0-6" class="anchor" href="#p-465594-tensor6-3-0-6-devicecuda0-torchsize4-class-torchtensor-tensor-00912-00718-11203-05582-02274-02479-01236-20472-15937-01362-04223-09636-03833-02258-03686-07650-09438-07804-01861-01010-06031-00587-00172-05022-04774-15771-15994-01907-00961-14858-01823-00920-04004-00362-18670-15711-devicecuda0-torchsize4-9-class-torchtensor-tensor21856-devicecuda0-epoch-11-validation-8-11132-00010014-845its-tensor6-1-6-6-devicecuda0-torchsize4-class-torchtensor-tensor-00159-03259-10007-01159-04197-09344-07106-19948-14186-05346-09564-02370-09762-07139-10262-07824-18359-23684-00397-04967-00271-02689-02430-02428-02138-22142-19273-02429-01047-05400-07540-01807-05147-02106-14485-17857-devicecuda0-torchsize4-9-class-torchtensor-tensor15966-devicecuda0-6"></a>tensor([6, 3, 0, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.0912, -0.0718, -1.1203,  0.5582,  0.2274, -0.2479,  0.1236, -2.0472,<br>
-1.5937],<br>
[-0.1362,  0.4223, -0.9636, -0.3833, -0.2258, -0.3686,  0.7650, -0.9438,<br>
-0.7804],<br>
[-0.1861, -0.1010, -0.6031, -0.0587, -0.0172,  0.5022,  0.4774, -1.5771,<br>
-1.5994],<br>
[ 0.1907, -0.0961, -1.4858,  0.1823,  0.0920,  0.4004,  0.0362, -1.8670,<br>
-1.5711]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.1856, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   8%|▊         | 11/132 [00:01&lt;00:14,  8.45it/s]----------------<br>
tensor([6, 1, 6, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.0159,  0.3259, -1.0007,  0.1159, -0.4197, -0.9344,  0.7106, -1.9948,<br>
-1.4186],<br>
[-0.5346,  0.9564, -0.2370,  0.9762, -0.7139, -1.0262,  0.7824, -1.8359,<br>
-2.3684],<br>
[-0.0397, -0.4967,  0.0271,  0.2689,  0.2430, -0.2428, -0.2138, -2.2142,<br>
-1.9273],<br>
[ 0.2429,  0.1047, -0.5400,  0.7540, -0.1807, -0.5147,  0.2106, -1.4485,<br>
-1.7857]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.5966, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor3-6-6-4-devicecuda0-torchsize4-class-torchtensor-tensor-00623-01720-02256-22174-12514-17831-00996-33510-32957-00397-04967-00271-02689-02430-02428-02138-22142-19273-02822-05983-06013-04433-07331-09755-09977-14601-17671-02910-09920-10511-06854-04868-25009-02337-42157-33742-devicecuda0-torchsize4-9-class-torchtensor-tensor17746-devicecuda0-epoch-11-validation-10-13132-00010013-858its-tensor3-6-6-2-devicecuda0-torchsize4-class-torchtensor-tensor-02128-00466-05491-07120-03784-06170-00678-24010-17028-01700-03103-11498-00080-01146-02749-05718-30480-21813-02715-05407-14621-08740-00778-05175-09095-27573-24112-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor27389-devicecuda0-7" class="anchor" href="#p-465594-tensor3-6-6-4-devicecuda0-torchsize4-class-torchtensor-tensor-00623-01720-02256-22174-12514-17831-00996-33510-32957-00397-04967-00271-02689-02430-02428-02138-22142-19273-02822-05983-06013-04433-07331-09755-09977-14601-17671-02910-09920-10511-06854-04868-25009-02337-42157-33742-devicecuda0-torchsize4-9-class-torchtensor-tensor17746-devicecuda0-epoch-11-validation-10-13132-00010013-858its-tensor3-6-6-2-devicecuda0-torchsize4-class-torchtensor-tensor-02128-00466-05491-07120-03784-06170-00678-24010-17028-01700-03103-11498-00080-01146-02749-05718-30480-21813-02715-05407-14621-08740-00778-05175-09095-27573-24112-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor27389-devicecuda0-7"></a>tensor([3, 6, 6, 4], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.0623, -0.1720, -0.2256,  2.2174, -1.2514, -1.7831, -0.0996, -3.3510,<br>
-3.2957],<br>
[-0.0397, -0.4967,  0.0271,  0.2689,  0.2430, -0.2428, -0.2138, -2.2142,<br>
-1.9273],<br>
[-0.2822,  0.5983, -0.6013,  0.4433, -0.7331, -0.9755,  0.9977, -1.4601,<br>
-1.7671],<br>
[-0.2910, -0.9920, -1.0511, -0.6854,  0.4868,  2.5009, -0.2337, -4.2157,<br>
-3.3742]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.7746, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:  10%|▉         | 13/132 [00:01&lt;00:13,  8.58it/s]----------------<br>
tensor([3, 6, 6, 2], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.2128,  0.0466, -0.5491,  0.7120, -0.3784, -0.6170,  0.0678, -2.4010,<br>
-1.7028],<br>
[ 0.1700, -0.3103, -1.1498,  0.0080, -0.1146,  0.2749,  0.5718, -3.0480,<br>
-2.1813],<br>
[-0.2715,  0.5407, -1.4621, -0.8740, -0.0778,  0.5175,  0.9095, -2.7573,<br>
-2.4112],<br>
[ 0.1521,  0.1529, -0.6599,  2.0708, -0.4169, -1.6593, -0.1911, -2.6210,<br>
-3.0652]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.7389, device=‘cuda:0’)</h2>
<h2><a name="p-465594-tensor6-6-3-6-devicecuda0-torchsize4-class-torchtensor-tensor-00039-04696-10225-14045-05902-11559-01638-26928-15331-00638-05005-11653-02601-03987-00643-08232-14107-13968-06122-02130-09526-00982-01198-01349-00865-15231-12064-03609-01723-09109-02720-01903-06936-03121-15757-13673-devicecuda0-torchsize4-9-class-torchtensor-tensor17441-devicecuda0-epoch-11-validation-11-14132-00010014-815its-8" class="anchor" href="#p-465594-tensor6-6-3-6-devicecuda0-torchsize4-class-torchtensor-tensor-00039-04696-10225-14045-05902-11559-01638-26928-15331-00638-05005-11653-02601-03987-00643-08232-14107-13968-06122-02130-09526-00982-01198-01349-00865-15231-12064-03609-01723-09109-02720-01903-06936-03121-15757-13673-devicecuda0-torchsize4-9-class-torchtensor-tensor17441-devicecuda0-epoch-11-validation-11-14132-00010014-815its-8"></a>tensor([6, 6, 3, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.0039, -0.4696, -1.0225, -1.4045,  0.5902,  1.1559,  0.1638, -2.6928,<br>
-1.5331],<br>
[ 0.0638,  0.5005, -1.1653, -0.2601, -0.3987,  0.0643,  0.8232, -1.4107,<br>
-1.3968],<br>
[ 0.6122, -0.2130, -0.9526,  0.0982, -0.1198, -0.1349,  0.0865, -1.5231,<br>
-1.2064],<br>
[ 0.3609,  0.1723, -0.9109,  0.2720, -0.1903, -0.6936,  0.3121, -1.5757,<br>
-1.3673]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.7441, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:  11%|█         | 14/132 [00:01&lt;00:14,  8.15it/s]</h2>
<p>tensor([0, 0, 5, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;</p>
<p>this is the full trace I added some print statements, not seeing anything like this. Thanks, any help will highly be appreciated</p>

ptrblck: <p>I don’t see an error so assume your run did not crash yet.</p>

Atharva_Mishra: <p>no same error is there I did’t posted again full trace with error:</p>
<h2><a name="p-465596-epoch-11-training-100-530530-00050000-9433its-epoch-11-validation-1-1132-00000021-620itstensor6-0-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01172-01543-05504-02245-00286-00923-03298-21161-18404-01884-09037-04673-00721-11703-06113-07804-28961-22404-03921-00559-13258-04566-02808-01708-01426-14904-16001-05576-02592-00398-03589-04208-00898-08281-21817-21289-devicecuda0-torchsize4-9-class-torchtensor-tensor20431-devicecuda0-1" class="anchor" href="#p-465596-epoch-11-training-100-530530-00050000-9433its-epoch-11-validation-1-1132-00000021-620itstensor6-0-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01172-01543-05504-02245-00286-00923-03298-21161-18404-01884-09037-04673-00721-11703-06113-07804-28961-22404-03921-00559-13258-04566-02808-01708-01426-14904-16001-05576-02592-00398-03589-04208-00898-08281-21817-21289-devicecuda0-torchsize4-9-class-torchtensor-tensor20431-devicecuda0-1"></a>Epoch 1/1 [Training]: 100%|██████████| 530/530 [00:05&lt;00:00, 94.33it/s]<br>
Epoch 1/1 [Validation]:   1%|          | 1/132 [00:00&lt;00:21,  6.20it/s]tensor([6, 0, 5, 2], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.1172,  0.1543, -0.5504,  0.2245, -0.0286,  0.0923,  0.3298, -2.1161,<br>
-1.8404],<br>
[ 0.1884, -0.9037, -0.4673, -0.0721,  1.1703,  0.6113, -0.7804, -2.8961,<br>
-2.2404],<br>
[ 0.3921, -0.0559, -1.3258,  0.4566,  0.2808, -0.1708, -0.1426, -1.4904,<br>
-1.6001],<br>
[-0.5576, -0.2592, -0.0398, -0.3589, -0.4208,  0.0898,  0.8281, -2.1817,<br>
-2.1289]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.0431, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor0-6-6-5-devicecuda0-torchsize4-class-torchtensor-tensor-03324-11683-09862-04847-17634-04665-15038-38269-27187-03151-08455-07605-01490-08000-09488-13387-14526-18561-02988-02303-00565-10323-02438-09100-03279-20378-25445-05214-11528-06574-00817-12476-07205-10407-33927-25857-devicecuda0-torchsize4-9-class-torchtensor-tensor16296-devicecuda0-epoch-11-validation-2-3132-00000016-787its-tensor5-6-3-0-devicecuda0-torchsize4-class-torchtensor-tensor-02032-04361-06183-18954-07922-14702-01195-24953-26830-02581-05257-10674-00423-01433-02777-03005-31134-22096-00335-02139-05577-03850-01559-05312-00724-08430-08123-01569-02349-06334-09193-02774-06634-04086-19756-10669-devicecuda0-torchsize4-9-class-torchtensor-tensor27672-devicecuda0-2" class="anchor" href="#p-465596-tensor0-6-6-5-devicecuda0-torchsize4-class-torchtensor-tensor-03324-11683-09862-04847-17634-04665-15038-38269-27187-03151-08455-07605-01490-08000-09488-13387-14526-18561-02988-02303-00565-10323-02438-09100-03279-20378-25445-05214-11528-06574-00817-12476-07205-10407-33927-25857-devicecuda0-torchsize4-9-class-torchtensor-tensor16296-devicecuda0-epoch-11-validation-2-3132-00000016-787its-tensor5-6-3-0-devicecuda0-torchsize4-class-torchtensor-tensor-02032-04361-06183-18954-07922-14702-01195-24953-26830-02581-05257-10674-00423-01433-02777-03005-31134-22096-00335-02139-05577-03850-01559-05312-00724-08430-08123-01569-02349-06334-09193-02774-06634-04086-19756-10669-devicecuda0-torchsize4-9-class-torchtensor-tensor27672-devicecuda0-2"></a>tensor([0, 6, 6, 5], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.3324, -1.1683, -0.9862,  0.4847,  1.7634,  0.4665, -1.5038, -3.8269,<br>
-2.7187],<br>
[-0.3151,  0.8455, -0.7605,  0.1490, -0.8000, -0.9488,  1.3387, -1.4526,<br>
-1.8561],<br>
[-0.2988,  0.2303,  0.0565,  1.0323, -0.2438, -0.9100,  0.3279, -2.0378,<br>
-2.5445],<br>
[ 0.5214, -1.1528, -0.6574,  0.0817,  1.2476,  0.7205, -1.0407, -3.3927,<br>
-2.5857]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.6296, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   2%|▏         | 3/132 [00:00&lt;00:16,  7.87it/s]----------------<br>
tensor([5, 6, 3, 0], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.2032,  0.4361, -0.6183,  1.8954, -0.7922, -1.4702, -0.1195, -2.4953,<br>
-2.6830],<br>
[ 0.2581, -0.5257, -1.0674,  0.0423,  0.1433,  0.2777,  0.3005, -3.1134,<br>
-2.2096],<br>
[ 0.0335,  0.2139, -0.5577,  0.3850, -0.1559, -0.5312,  0.0724, -0.8430,<br>
-0.8123],<br>
[ 0.1569, -0.2349, -0.6334, -0.9193,  0.2774,  0.6634,  0.4086, -1.9756,<br>
-1.0669]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.7672, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor5-4-3-3-devicecuda0-torchsize4-class-torchtensor-tensor-07228-13446-09910-01117-12572-12824-12669-39945-28691-01884-09037-04673-00721-11703-06113-07804-28961-22404-04943-09349-08567-02696-10498-05423-07203-25375-18992-04857-02578-10885-15036-04540-06005-09201-32721-27364-devicecuda0-torchsize4-9-class-torchtensor-tensor12083-devicecuda0-epoch-11-validation-4-5132-00000016-776its-tensor1-6-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01827-04180-10472-01140-06313-00492-08390-25656-17660-04286-08027-04469-14552-05709-08960-03469-32073-28408-02910-09920-10511-06854-04868-25009-02337-42157-33742-01958-04928-00435-00273-00136-00224-01098-21593-19146-devicecuda0-torchsize4-9-class-torchtensor-tensor15772-devicecuda0-3" class="anchor" href="#p-465596-tensor5-4-3-3-devicecuda0-torchsize4-class-torchtensor-tensor-07228-13446-09910-01117-12572-12824-12669-39945-28691-01884-09037-04673-00721-11703-06113-07804-28961-22404-04943-09349-08567-02696-10498-05423-07203-25375-18992-04857-02578-10885-15036-04540-06005-09201-32721-27364-devicecuda0-torchsize4-9-class-torchtensor-tensor12083-devicecuda0-epoch-11-validation-4-5132-00000016-776its-tensor1-6-5-2-devicecuda0-torchsize4-class-torchtensor-tensor-01827-04180-10472-01140-06313-00492-08390-25656-17660-04286-08027-04469-14552-05709-08960-03469-32073-28408-02910-09920-10511-06854-04868-25009-02337-42157-33742-01958-04928-00435-00273-00136-00224-01098-21593-19146-devicecuda0-torchsize4-9-class-torchtensor-tensor15772-devicecuda0-3"></a>tensor([5, 4, 3, 3], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.7228, -1.3446, -0.9910, -0.1117,  1.2572,  1.2824, -1.2669, -3.9945,<br>
-2.8691],<br>
[ 0.1884, -0.9037, -0.4673, -0.0721,  1.1703,  0.6113, -0.7804, -2.8961,<br>
-2.2404],<br>
[ 0.4943, -0.9349, -0.8567, -0.2696,  1.0498,  0.5423, -0.7203, -2.5375,<br>
-1.8992],<br>
[ 0.4857, -0.2578, -1.0885,  1.5036,  0.4540, -0.6005, -0.9201, -3.2721,<br>
-2.7364]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.2083, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   4%|▍         | 5/132 [00:00&lt;00:16,  7.76it/s]----------------<br>
tensor([1, 6, 5, 2], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.1827,  0.4180, -1.0472, -0.1140, -0.6313, -0.0492,  0.8390, -2.5656,<br>
-1.7660],<br>
[ 0.4286, -0.8027, -0.4469,  1.4552, -0.5709, -0.8960, -0.3469, -3.2073,<br>
-2.8408],<br>
[-0.2910, -0.9920, -1.0511, -0.6854,  0.4868,  2.5009, -0.2337, -4.2157,<br>
-3.3742],<br>
[-0.1958, -0.4928,  0.0435, -0.0273,  0.0136,  0.0224,  0.1098, -2.1593,<br>
-1.9146]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.5772, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor6-5-6-3-devicecuda0-torchsize4-class-torchtensor-tensor-02412-06586-09805-08835-05440-09555-01473-24310-15796-02910-09920-10511-06854-04868-25009-02337-42157-33742-02032-04361-06183-18954-07922-14702-01195-24953-26830-11571-08971-11486-17376-00474-05935-13113-37319-32834-devicecuda0-torchsize4-9-class-torchtensor-tensor09657-devicecuda0-epoch-11-validation-5-7132-00000015-814its-tensor0-2-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-11050-10732-13967-16984-01679-01173-12775-47368-40365-03627-00192-04877-08786-00692-06484-00771-17631-22243-04638-06656-06289-17969-01436-07338-07436-36590-32770-01481-07261-11944-01191-02490-05490-06908-16781-17578-devicecuda0-torchsize4-9-class-torchtensor-tensor21222-devicecuda0-4" class="anchor" href="#p-465596-tensor6-5-6-3-devicecuda0-torchsize4-class-torchtensor-tensor-02412-06586-09805-08835-05440-09555-01473-24310-15796-02910-09920-10511-06854-04868-25009-02337-42157-33742-02032-04361-06183-18954-07922-14702-01195-24953-26830-11571-08971-11486-17376-00474-05935-13113-37319-32834-devicecuda0-torchsize4-9-class-torchtensor-tensor09657-devicecuda0-epoch-11-validation-5-7132-00000015-814its-tensor0-2-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-11050-10732-13967-16984-01679-01173-12775-47368-40365-03627-00192-04877-08786-00692-06484-00771-17631-22243-04638-06656-06289-17969-01436-07338-07436-36590-32770-01481-07261-11944-01191-02490-05490-06908-16781-17578-devicecuda0-torchsize4-9-class-torchtensor-tensor21222-devicecuda0-4"></a>tensor([6, 5, 6, 3], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.2412, -0.6586, -0.9805, -0.8835,  0.5440,  0.9555,  0.1473, -2.4310,<br>
-1.5796],<br>
[-0.2910, -0.9920, -1.0511, -0.6854,  0.4868,  2.5009, -0.2337, -4.2157,<br>
-3.3742],<br>
[ 0.2032,  0.4361, -0.6183,  1.8954, -0.7922, -1.4702, -0.1195, -2.4953,<br>
-2.6830],<br>
[ 1.1571, -0.8971, -1.1486,  1.7376,  0.0474, -0.5935, -1.3113, -3.7319,<br>
-3.2834]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(0.9657, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   5%|▌         | 7/132 [00:00&lt;00:15,  8.14it/s]----------------<br>
tensor([0, 2, 3, 1], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 1.1050, -1.0732, -1.3967,  1.6984,  0.1679, -0.1173, -1.2775, -4.7368,<br>
-4.0365],<br>
[ 0.3627, -0.0192, -0.4877,  0.8786, -0.0692, -0.6484,  0.0771, -1.7631,<br>
-2.2243],<br>
[ 0.4638, -0.6656, -0.6289,  1.7969, -0.1436, -0.7338, -0.7436, -3.6590,<br>
-3.2770],<br>
[-0.1481,  0.7261, -1.1944,  0.1191, -0.2490, -0.5490,  0.6908, -1.6781,<br>
-1.7578]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.1222, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor1-5-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-02084-03109-05869-03480-02416-00612-03557-07210-05134-00914-04602-11326-16102-07496-14515-02776-35171-20348-02383-02433-07864-13938-02066-05387-00077-24466-29635-01002-00886-10583-03418-03617-02141-05558-15194-12726-devicecuda0-torchsize4-9-class-torchtensor-tensor12938-devicecuda0-epoch-11-validation-7-9132-00010014-832its-tensor1-0-0-0-devicecuda0-torchsize4-class-torchtensor-tensor-00827-09098-15061-00083-05200-08133-12496-13435-15232-00197-00608-01855-03570-01213-07042-00125-14624-12429-12857-13768-15441-18789-01597-00293-17447-50522-41930-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor17246-devicecuda0-5" class="anchor" href="#p-465596-tensor1-5-3-1-devicecuda0-torchsize4-class-torchtensor-tensor-02084-03109-05869-03480-02416-00612-03557-07210-05134-00914-04602-11326-16102-07496-14515-02776-35171-20348-02383-02433-07864-13938-02066-05387-00077-24466-29635-01002-00886-10583-03418-03617-02141-05558-15194-12726-devicecuda0-torchsize4-9-class-torchtensor-tensor12938-devicecuda0-epoch-11-validation-7-9132-00010014-832its-tensor1-0-0-0-devicecuda0-torchsize4-class-torchtensor-tensor-00827-09098-15061-00083-05200-08133-12496-13435-15232-00197-00608-01855-03570-01213-07042-00125-14624-12429-12857-13768-15441-18789-01597-00293-17447-50522-41930-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor17246-devicecuda0-5"></a>tensor([1, 5, 3, 1], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.2084,  0.3109, -0.5869, -0.3480, -0.2416, -0.0612,  0.3557, -0.7210,<br>
-0.5134],<br>
[-0.0914, -0.4602, -1.1326, -1.6102,  0.7496,  1.4515,  0.2776, -3.5171,<br>
-2.0348],<br>
[ 0.2383, -0.2433, -0.7864,  1.3938, -0.2066, -0.5387, -0.0077, -2.4466,<br>
-2.9635],<br>
[ 0.1002,  0.0886, -1.0583, -0.3418, -0.3617,  0.2141,  0.5558, -1.5194,<br>
-1.2726]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.2938, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   7%|▋         | 9/132 [00:01&lt;00:14,  8.32it/s]----------------<br>
tensor([1, 0, 0, 0], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.0827,  0.9098, -1.5061,  0.0083, -0.5200, -0.8133,  1.2496, -1.3435,<br>
-1.5232],<br>
[-0.0197, -0.0608, -0.1855,  0.3570, -0.1213, -0.7042,  0.0125, -1.4624,<br>
-1.2429],<br>
[ 1.2857, -1.3768, -1.5441,  1.8789,  0.1597, -0.0293, -1.7447, -5.0522,<br>
-4.1930],<br>
[ 0.1521,  0.1529, -0.6599,  2.0708, -0.4169, -1.6593, -0.1911, -2.6210,<br>
-3.0652]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.7246, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor6-3-0-6-devicecuda0-torchsize4-class-torchtensor-tensor-00912-00718-11203-05582-02274-02479-01236-20472-15937-01362-04223-09636-03833-02258-03686-07650-09438-07804-01861-01010-06031-00587-00172-05022-04774-15771-15994-01907-00961-14858-01823-00920-04004-00362-18670-15711-devicecuda0-torchsize4-9-class-torchtensor-tensor21856-devicecuda0-epoch-11-validation-8-11132-00010014-845its-tensor6-1-6-6-devicecuda0-torchsize4-class-torchtensor-tensor-00159-03259-10007-01159-04197-09344-07106-19948-14186-05346-09564-02370-09762-07139-10262-07824-18359-23684-00397-04967-00271-02689-02430-02428-02138-22142-19273-02429-01047-05400-07540-01807-05147-02106-14485-17857-devicecuda0-torchsize4-9-class-torchtensor-tensor15966-devicecuda0-6" class="anchor" href="#p-465596-tensor6-3-0-6-devicecuda0-torchsize4-class-torchtensor-tensor-00912-00718-11203-05582-02274-02479-01236-20472-15937-01362-04223-09636-03833-02258-03686-07650-09438-07804-01861-01010-06031-00587-00172-05022-04774-15771-15994-01907-00961-14858-01823-00920-04004-00362-18670-15711-devicecuda0-torchsize4-9-class-torchtensor-tensor21856-devicecuda0-epoch-11-validation-8-11132-00010014-845its-tensor6-1-6-6-devicecuda0-torchsize4-class-torchtensor-tensor-00159-03259-10007-01159-04197-09344-07106-19948-14186-05346-09564-02370-09762-07139-10262-07824-18359-23684-00397-04967-00271-02689-02430-02428-02138-22142-19273-02429-01047-05400-07540-01807-05147-02106-14485-17857-devicecuda0-torchsize4-9-class-torchtensor-tensor15966-devicecuda0-6"></a>tensor([6, 3, 0, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.0912, -0.0718, -1.1203,  0.5582,  0.2274, -0.2479,  0.1236, -2.0472,<br>
-1.5937],<br>
[-0.1362,  0.4223, -0.9636, -0.3833, -0.2258, -0.3686,  0.7650, -0.9438,<br>
-0.7804],<br>
[-0.1861, -0.1010, -0.6031, -0.0587, -0.0172,  0.5022,  0.4774, -1.5771,<br>
-1.5994],<br>
[ 0.1907, -0.0961, -1.4858,  0.1823,  0.0920,  0.4004,  0.0362, -1.8670,<br>
-1.5711]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.1856, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:   8%|▊         | 11/132 [00:01&lt;00:14,  8.45it/s]----------------<br>
tensor([6, 1, 6, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.0159,  0.3259, -1.0007,  0.1159, -0.4197, -0.9344,  0.7106, -1.9948,<br>
-1.4186],<br>
[-0.5346,  0.9564, -0.2370,  0.9762, -0.7139, -1.0262,  0.7824, -1.8359,<br>
-2.3684],<br>
[-0.0397, -0.4967,  0.0271,  0.2689,  0.2430, -0.2428, -0.2138, -2.2142,<br>
-1.9273],<br>
[ 0.2429,  0.1047, -0.5400,  0.7540, -0.1807, -0.5147,  0.2106, -1.4485,<br>
-1.7857]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.5966, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor3-6-6-4-devicecuda0-torchsize4-class-torchtensor-tensor-00623-01720-02256-22174-12514-17831-00996-33510-32957-00397-04967-00271-02689-02430-02428-02138-22142-19273-02822-05983-06013-04433-07331-09755-09977-14601-17671-02910-09920-10511-06854-04868-25009-02337-42157-33742-devicecuda0-torchsize4-9-class-torchtensor-tensor17746-devicecuda0-epoch-11-validation-10-13132-00010013-858its-tensor3-6-6-2-devicecuda0-torchsize4-class-torchtensor-tensor-02128-00466-05491-07120-03784-06170-00678-24010-17028-01700-03103-11498-00080-01146-02749-05718-30480-21813-02715-05407-14621-08740-00778-05175-09095-27573-24112-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor27389-devicecuda0-7" class="anchor" href="#p-465596-tensor3-6-6-4-devicecuda0-torchsize4-class-torchtensor-tensor-00623-01720-02256-22174-12514-17831-00996-33510-32957-00397-04967-00271-02689-02430-02428-02138-22142-19273-02822-05983-06013-04433-07331-09755-09977-14601-17671-02910-09920-10511-06854-04868-25009-02337-42157-33742-devicecuda0-torchsize4-9-class-torchtensor-tensor17746-devicecuda0-epoch-11-validation-10-13132-00010013-858its-tensor3-6-6-2-devicecuda0-torchsize4-class-torchtensor-tensor-02128-00466-05491-07120-03784-06170-00678-24010-17028-01700-03103-11498-00080-01146-02749-05718-30480-21813-02715-05407-14621-08740-00778-05175-09095-27573-24112-01521-01529-06599-20708-04169-16593-01911-26210-30652-devicecuda0-torchsize4-9-class-torchtensor-tensor27389-devicecuda0-7"></a>tensor([3, 6, 6, 4], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.0623, -0.1720, -0.2256,  2.2174, -1.2514, -1.7831, -0.0996, -3.3510,<br>
-3.2957],<br>
[-0.0397, -0.4967,  0.0271,  0.2689,  0.2430, -0.2428, -0.2138, -2.2142,<br>
-1.9273],<br>
[-0.2822,  0.5983, -0.6013,  0.4433, -0.7331, -0.9755,  0.9977, -1.4601,<br>
-1.7671],<br>
[-0.2910, -0.9920, -1.0511, -0.6854,  0.4868,  2.5009, -0.2337, -4.2157,<br>
-3.3742]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.7746, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:  10%|▉         | 13/132 [00:01&lt;00:13,  8.58it/s]----------------<br>
tensor([3, 6, 6, 2], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[ 0.2128,  0.0466, -0.5491,  0.7120, -0.3784, -0.6170,  0.0678, -2.4010,<br>
-1.7028],<br>
[ 0.1700, -0.3103, -1.1498,  0.0080, -0.1146,  0.2749,  0.5718, -3.0480,<br>
-2.1813],<br>
[-0.2715,  0.5407, -1.4621, -0.8740, -0.0778,  0.5175,  0.9095, -2.7573,<br>
-2.4112],<br>
[ 0.1521,  0.1529, -0.6599,  2.0708, -0.4169, -1.6593, -0.1911, -2.6210,<br>
-3.0652]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(2.7389, device=‘cuda:0’)</h2>
<h2><a name="p-465596-tensor6-6-3-6-devicecuda0-torchsize4-class-torchtensor-tensor-00039-04696-10225-14045-05902-11559-01638-26928-15331-00638-05005-11653-02601-03987-00643-08232-14107-13968-06122-02130-09526-00982-01198-01349-00865-15231-12064-03609-01723-09109-02720-01903-06936-03121-15757-13673-devicecuda0-torchsize4-9-class-torchtensor-tensor17441-devicecuda0-epoch-11-validation-11-14132-00010014-815its-8" class="anchor" href="#p-465596-tensor6-6-3-6-devicecuda0-torchsize4-class-torchtensor-tensor-00039-04696-10225-14045-05902-11559-01638-26928-15331-00638-05005-11653-02601-03987-00643-08232-14107-13968-06122-02130-09526-00982-01198-01349-00865-15231-12064-03609-01723-09109-02720-01903-06936-03121-15757-13673-devicecuda0-torchsize4-9-class-torchtensor-tensor17441-devicecuda0-epoch-11-validation-11-14132-00010014-815its-8"></a>tensor([6, 6, 3, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor([[-0.0039, -0.4696, -1.0225, -1.4045,  0.5902,  1.1559,  0.1638, -2.6928,<br>
-1.5331],<br>
[ 0.0638,  0.5005, -1.1653, -0.2601, -0.3987,  0.0643,  0.8232, -1.4107,<br>
-1.3968],<br>
[ 0.6122, -0.2130, -0.9526,  0.0982, -0.1198, -0.1349,  0.0865, -1.5231,<br>
-1.2064],<br>
[ 0.3609,  0.1723, -0.9109,  0.2720, -0.1903, -0.6936,  0.3121, -1.5757,<br>
-1.3673]], device=‘cuda:0’)<br>
torch.Size([4, 9])<br>
&lt;class ‘torch.Tensor’&gt;<br>
tensor(1.7441, device=‘cuda:0’)<br>
Epoch 1/1 [Validation]:  11%|█         | 14/132 [00:01&lt;00:14,  8.15it/s]</h2>
<p>tensor([0, 0, 5, 6], device=‘cuda:0’)<br>
torch.Size([4])<br>
&lt;class ‘torch.Tensor’&gt;</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[18], line 58<br>
55 print(type(labels))<br>
57 outputs = model(text_input.long(), gene_input.long(), variance_input.long())<br>
—&gt; 58 print(outputs)<br>
59 print(outputs.shape)<br>
60 print(type(outputs))</p>
<p>File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch_tensor.py:568, in Tensor.<strong>repr</strong>(self, tensor_contents)<br>
564     return handle_torch_function(<br>
565         Tensor.<strong>repr</strong>, (self,), self, tensor_contents=tensor_contents<br>
566     )<br>
567 # All strings are unicode in Python 3.<br>
 → 568 return torch._tensor_str._str(self, tensor_contents=tensor_contents)</p>
<p>File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch_tensor_str.py:704, in _str(self, tensor_contents)<br>
702 with torch.no_grad(), torch.utils._python_dispatch._disable_current_modes():<br>
703     guard = torch._C._DisableFuncTorch()<br>
 → 704     return _str_intern(self, tensor_contents=tensor_contents)</p>
<p>File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch_tensor_str.py:621, in _str_intern(inp, tensor_contents)<br>
619                     tensor_str = _tensor_str(self.to_dense(), indent)<br>
620                 else:<br>
 → 621                     tensor_str = _tensor_str(self, indent)<br>
623 if self.layout != torch.strided:<br>
624     suffixes.append(“layout=” + str(self.layout))</p>
<p>File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch_tensor_str.py:353, in _tensor_str(self, indent)<br>
349     return _tensor_str_with_formatter(<br>
350         self, indent, summarize, real_formatter, imag_formatter<br>
351     )<br>
352 else:<br>
 → 353     formatter = _Formatter(get_summarized_data(self) if summarize else self)<br>
354     return _tensor_str_with_formatter(self, indent, summarize, formatter)</p>
<p>File u:\nlp_project\Personalized-Medicine-Redefining-Cancer-Treatment\venv\lib\site-packages\torch_tensor_str.py:145, in _Formatter.<strong>init</strong>(self, tensor)<br>
142         self.max_width = max(self.max_width, len(value_str))<br>
144 else:<br>
 → 145     nonzero_finite_vals = torch.masked_select(<br>
146         tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0)<br>
147     )<br>
149     if nonzero_finite_vals.numel() == 0:<br>
150         # no valid number, do nothing<br>
151         return</p>
<p>RuntimeError: CUDA error: device-side assert triggered<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>